1. This isn't me disagreeing with you, Émile, but while you are correct that many, many people who subscribe to the TESCREAL worldview do genuinely believe that a "Misaligned AI" could/will be coming soon and WILL kill all of Humanity...I think it's still important to remember that these people, from Yud to Altman to Sutskever and more, are most likely suffering from some kind of AI Psychosis. You say it yourself; they're "Religious Extremists". There's a LOT of people who also believe in the Rapture. And yet the Rapture has never happened despite the crying, preaching, and constant media attention from these Religious Extremists. I just think that while this talk of Misaligned AI is scary...we have to remember (Which you already do :) that we are hearing this from people who are Techno-Religious Extremists...Who also think Humanity should die. That's bad, but they don't need an AI God to do that; they can just kill us with pollution from Data Centers.
2. All this talk with Jacob Coxon has got me thinking, as I usually do, about something in particular: Does anyone remember Daniel Kokotaijlo, the guy who made AI2027? He and Jacob Coxon are...basically the same thing. Sure, Kokotaijlo didn't make a Tweet saying AI could kill us by the end of the decade and that AI Companies need to "Lock in and make a Kill Switch", but he did make AI2027, which, while terrible and mostly a hypothetical Sci-Fi, anti-China PowerPoint, was talked about for a while. Even JD Vance "claimed" to read it. Kokotaijlo also got big because he claimed not to have signed an NDA with OpenAI so as to bash them (Still retained equity, though).
My point is, why has the news reacted this way with Coxon, when Kokotaijlo literally did the same thing LAST YEAR!?
The thought of “posthumans living in computer simulations” is about as horrific a vision of the future as I can imagine. If that came to pass, the title of that paper you wrote (“Do We Live in Hell?”) would no longer be rhetorical—we really would be living in literal hell!
Right. The thought of it itself has given me already two horrendus dreams. It's a crazy level of fear of death mixed with a lack of imagination.
I fear death. I've learnt to accept that fear and even find comfort in it. After all a fear of death is, often (but not always), an affirmation of life.
But there are so so so many things worse than death.
Yes, a fear of death for sure. Also, extreme alienation from nature and a lack of appreciation for the beauty of biological life. Didn’t these people ever take a humanities class, or go for a walk in the woods?
Two things I wanted to say:
1. This isn't me disagreeing with you, Émile, but while you are correct that many, many people who subscribe to the TESCREAL worldview do genuinely believe that a "Misaligned AI" could/will be coming soon and WILL kill all of Humanity...I think it's still important to remember that these people, from Yud to Altman to Sutskever and more, are most likely suffering from some kind of AI Psychosis. You say it yourself; they're "Religious Extremists". There's a LOT of people who also believe in the Rapture. And yet the Rapture has never happened despite the crying, preaching, and constant media attention from these Religious Extremists. I just think that while this talk of Misaligned AI is scary...we have to remember (Which you already do :) that we are hearing this from people who are Techno-Religious Extremists...Who also think Humanity should die. That's bad, but they don't need an AI God to do that; they can just kill us with pollution from Data Centers.
2. All this talk with Jacob Coxon has got me thinking, as I usually do, about something in particular: Does anyone remember Daniel Kokotaijlo, the guy who made AI2027? He and Jacob Coxon are...basically the same thing. Sure, Kokotaijlo didn't make a Tweet saying AI could kill us by the end of the decade and that AI Companies need to "Lock in and make a Kill Switch", but he did make AI2027, which, while terrible and mostly a hypothetical Sci-Fi, anti-China PowerPoint, was talked about for a while. Even JD Vance "claimed" to read it. Kokotaijlo also got big because he claimed not to have signed an NDA with OpenAI so as to bash them (Still retained equity, though).
My point is, why has the news reacted this way with Coxon, when Kokotaijlo literally did the same thing LAST YEAR!?
Great post as always, Émile.
"They think that if we build an ASI that is “aligned,” it will grant them eternal life, enabling them to upload their minds to computers, etc."
This bit confuses me. Why on earth would you assume that uploading ones mind onto a computer would grant you eternal life?
It seems based on some notion that conciousness is a form of software. But doesn't seem an awful lot of evidence for that. Speculative at best.
Obviously, this is assuming that such uploading is even possible.
100% agree!
The thought of “posthumans living in computer simulations” is about as horrific a vision of the future as I can imagine. If that came to pass, the title of that paper you wrote (“Do We Live in Hell?”) would no longer be rhetorical—we really would be living in literal hell!
Right. The thought of it itself has given me already two horrendus dreams. It's a crazy level of fear of death mixed with a lack of imagination.
I fear death. I've learnt to accept that fear and even find comfort in it. After all a fear of death is, often (but not always), an affirmation of life.
But there are so so so many things worse than death.
Yes, a fear of death for sure. Also, extreme alienation from nature and a lack of appreciation for the beauty of biological life. Didn’t these people ever take a humanities class, or go for a walk in the woods?
He is a transhumanist theorist who has written extensively about the "transition" process. Maybe worth checking out
Ah, fascinating. Thanks for apprising me of this! Anything I should read in particular? A link or title? :-) Thanks again!
So how do we stop this? I feel despair, and that isn't going to change anything. What are some things we can do right now to prevent this catastrophe?
I am curious if you are familiar with Bryant McGill, and, if so, how he fits into this ASI story.
No, I'm not. Who is he? :-)