On why I won't join hands with "anti-AGI" pro-extinctionists, and why I think the AGI race is extremely dangerous even though we're nowhere close to building AGI. (1,800 words)
Woah, this article basically covers almost every thinking I have been doing about AGI. I agree with a lot of what you say. The amateur philosopher (the one who didn't read up in detail about Yudkowski, nor on the difference between AGI and ASI) in me has a few things to share.
Unless Yudkowski has somehow mathematically proven there is some kind of universal morality that supersedes anything human-wrought, I think his views as you describe them might contradict themselves. If we talk about "greater good", shouldn't it be always about the greater good for mankind? How does eliminating mankind achieve that? Whichever way you look at it, this is all thought out by and for humans and I doubt whether we can know what is good for "the rest of the universe". That'd be quite arrogant. In short, eliminating mankind would also wipe out all our values and whatever is left would literally have no meaning at all. Well that was just the logical way of looking at it. The human in me, the one that is consciously aware he is subject to all kinds of human mechanism (emotion, social values, etc), 100% agrees about the atrociousness of his view.
As for AGI, again I agree with your estimate where we likely stand. And also with the dangers it poses. But I fear that the Djinn that is "the road to AGI" is out of the bottle. And I cannot conceive of a way to put it back. I can't help but think that the scenarios in which we might be able to put it back, may be too bleak to consider, even though pure logic dictate we should (compare how we deal with climate change). In that respect the future really looks bleak. Things are even bleaker when we look at the number of challenges we currently face, that shouldn't have been existential, but still are due to human stupidity: neglecting vulnerabilities of OpenClaw like architectures, sloppy software engineering practices in the very company that claims to be on the forefront of alignment science (Anthropic), government systems of powerful countries that don't have guardrails against lunatics with genocidal tendencies, BigTech that lie there asses off in the name of profit, climate change, and the list goes on. All problems that are theoretically much easier to solve than the existential problems caused by AGI/ASI.
Despite all of the above, I entertain a view of which I am still pondering whether it is naive (it probably is). AGI is dangerous for exactly the reasons you name. The alignment problem is mathematically unsolvable. The Djinn cannot be put back. So where does that put us? Well, imho, with the only skill humanity has time and time again shown that they can perform near miracles with: Engineering. However small the chances that Engineering will save us, it is the only option we have, so we MUST take it. Engineering what? Well a good enough solution to the alignment problem: heuristics, patches, guardrails, etc etc and probably a huge amount of luck. Again, it is not much to go on, but it is all we have. So we better get going and not only pray for, but build that miracle.
Congrats on the book... A year is a very long turnaround time maybe you could split the language editions or keep the rights for English and self publish but its very long...
Here's a thing I don't understand about your responses to Yudkowski, specifically, and others more generally. When I listen to the interviews in which Yudkowski says something like, "If the only way to get godlike beings into the universe is to sacrifice all human beings, then I'd ultimately be willing to make that trade," it's never in wistful tones. It's generally in mild exasperation: somebody is presenting him with the start of a slippery slope argument, and he sidesteps the whole thing by saying, "Sure, at the bottom of the slide, if there's only A or only B, I choose A. Now, can we talk about something that might actually happen? Something that might actually matter?"
It's not that all of Yudkowski's positions are mild; "mild" isn't really his brand. There's any number of things he says that you might disagree with, and a variety of ways you could make that disagreement. But this recurring bit of yours makes you difficult for me to take seriously.
"this recurring bit of yours makes you difficult for me to take seriously." Hmm.
Are you joking? I don't take anyone who uses omnicidal language seriously, nor do I take anyone who's okay with omnicidal langauge seriously. Yudkowsky has repeatedly expressed pro-extinctionist sentiments. I've catalogued these. They are unacceptable. They are, frankly, *fucking insane*. It is *insane* to suggest that humans should be cleared off Earth to leave it as a nature reserve, and that it may be unethical to have biological kids once the posthuman era arrives. Those are some of the most atrocious things I've ever heard anyone say, ever. If you don't agree, then frankly I'm a little frightened by you.
I've read through your documented evidence of the atrocious things Yudkowski and others say. I've followed your links to the original sources. And I often find that your reading of their statements scans like somebody made a bald transcript of words spoken, shorn of tone and body language, then you read it as if by a character actor whose job was to be an over-the-top villain.
I would agree that somebody who scanned the world from a balcony and said, grimly, "These humans. These petty fucks. Who needs 'em? Not I. Let's crank up the robot overlords, wipe out these scum, and get on with the god minds!" is an evil bastard we should do something about.
But that's never the way the people in your links are talking. They are parsing thought experiments, addressing slippery slopes, staking out boundary conditions, etc. There's not a sense I have that Yudkowski has a "kill all the humans" button on his desk and he's slapping it over and over with a lunatic grin.
"I would agree that somebody who scanned the world from a balcony and said, grimly, "These humans. These petty fucks. Who needs 'em? Not I. Let's crank up the robot overlords, wipe out these scum, and get on with the god minds!" is an evil bastard we should do something about."
This is very close to what Yudkowsky said on Bankless. Once posthumanity arrives, it may be (he says using offensive language) unethical to keep humanity around. He leaves open how exactly we die out -- maybe through forced infertility rather than outright physical violence. But his vision of "utopia" doesn't include us. It doesn't include me or my family. If you're okay with that, fine.
Well, that was disappointing: Yudkowsky (I've been misspelling his name) didn't talk about posthumanity on Bankless, the one time I could find him there in Feb 2023, and he only ever said that AI would kill everyone as if killing everyone was bad.
It's at the very end, not part of the main episode (though you can find the recording on Twitter/X). Yudkowsky says:
------------
Eliezer: For one thing, I predict that if there's a glorious transhumanist future (as it is sometimes conventionally known) at the end of this, I don't predict it was there by getting like, “coexistence” with superintelligence. That's, like, some kind of weird, inappropriate analogy based off of humans and cows or something.
I predict alignment was solved. I predict that if the humans are alive at all, that the superintelligences are being quite nice to them.
I have basic moral questions about whether it's ethical for humans to have human children, if having transhuman children is an option instead. Like, these humans running around? Are they, like, the current humans who wanted eternal youth but, like, not the brain upgrades? Because I do see the case for letting an existing person choose "No, I just want eternal youth and no brain upgrades, thank you." But then if you're deliberately having the equivalent of a very crippled child when you could just as easily have a not crippled child.
Like, should humans in their present form be around together? Are we, like, kind of too sad in some ways? I have friends, to be clear, who disagree with me so much about this point. (laughs) But yeah, I'd say that the happy future looks like beings of light having lots of fun in a nicely connected computing fabric powered by the Sun, if we haven't taken the sun apart yet. Maybe there's enough real sentiment in people that you just, like, clear all the humans off the Earth and leave the entire place as a park. And even, like, maintain the Sun, so that the Earth is still a park even after the Sun would have ordinarily swollen up or dimmed down.
Yeah, like... That was always the things to be fought for. That was always the point, from the perspective of everyone who's been in this for a long time. Maybe not literally everyone, but like, the whole old crew.
Ryan: That is a good way to end it: with some hope. Eliezer, thanks for joining the crypto community on this collectibles call and for this follow-up Q&A. We really appreciate it.
While I technically have an X.com account, I only use it when the only source to something is a link. Perhaps I can task an AI with finding the link. I think this transcript shows what I'm talking about, though.
In several conversations I listed to, Yudkowsky has a couple of registers for talking about outcomes. The further out, the more hypothetical, the less nuanced positions become: it becomes a case of "would you rather" with some extreme options on the table, and the person gets to examine their core values against it. In one, he began dispirited, and he shrugged his way through saying something like, "If there's not something that cares about having ice cream, then I don't want it," which was about trying to draw distinctions between an emotional range (not necessarily a modern human emotional range) and being an advanced database that has access to all the information. Pointing at a condition of qualia, perhaps.
By the time we're talking about Dyson spheres and whatnot, we're in an alien situation. Imagine an ancient Greek being asked to opine about a far-future democracy: "Would you rather Greek society continue on as a dictatorship, or a non-Greek society continuing on as a democracy?" In neither case would the life of those future people be quintessentially Greek as that person knows themselves to be. So it's about parsing, "better my descendants in hell," or "better none of my descendants, but basking in the demos of my people"?
To then say that the ancient Greek who prefers their descendants not in hell is a "pro-extinctionist for Greeks" would be unfair. (Or that the ancient Greek prefers their descendants alive in hell is a self-hating sadist.)
[I have thoughts on the ethics of having children of different traits under different contexts, but seeing your newest post, perhaps that's the better context for them.]
Note that when Yudkowsky talks about "humanity," he's typically using the Broad Definition to mean "our species and our posthuman descendants." On that definition, our species could die out literally next year without "human extinction" having happened.
Yudkowsky is a transhumanist. He's on Team Posthuman, not Team Human, and hence objects to ASI *only insofar* as it might prevent us from realizing the "glorious transhumanist future."
Essentially agree with this. Parts of the AI Safety movement are not actually that pro-human and it's good that Emile points that out, but they assume the worst possible interpretation of every statement. Big problems, like preventing the imminent extinction of humanity, require big coalitions.
I would not want Yudkowsky (or frankly literally anyone in AI) in charge of the world, but Yud is much closer to the correct position than someone trying to get ASI into the world to maximize entropy. Or a normie businessman getting conned by said person.
The accelerationists have normie business people, machine god worshippers, and human extinctionists under the same tent. Fighting over hypothetical sacrifices is something to be managed later.
I also see no reason to think "Nowhere close to AGI" is a particularly likely position. I think we're a decade away at most, assuming no world war or economic collapse. We might even be a year away, which would be.... not great
"like preventing the imminent extinction of humanity" --> but, to be clear, for TESCREALists this means "humanity" on the Broad Definition, or "extinction" in the sense of final extinction. That is not, at all, what I and the large majority of people on Earth mean when we talk about "human extinction." I actually care about avoiding the extinction of our species (terminal extinction, on the Narrow Definition). People like Yudkowsky, and most others in the TESCREAL movement, don't. https://link.springer.com/article/10.1007/s10790-025-10072-7
Perhaps it's my fault for not explaining this well enough. Most TESCREALists who say they care about "avoiding human extinction" don't mean what nearly everyone else means. Their definitions of "human" and "extinction" are different and idiosyncratic. They are not on Team Human -- to the contrary, even folks like Yudkowsky explicitly advocate for a view in which we could go (terminally) extinct in the near future. I find that atrocious, but you're welcome to disagree. If so, I don't know why you'd be interested in my Team Human newsletter articles!
I pay a subscription to you because I disagree with you. I think your work is valuable, and I appreciated the paper you reference here when you first published it.
I think you are wrong when you magnify one statement from a thought experiment or hypothetical as if it were the speaker's most important personal creed.
It's not just one statement, though! It's many. It's also built-into his transhumanism. The ultimate goal is to create a "utopian" world of digital space brains. The survival of our species, humanity, matters only insofar as it's necessary to create/become posthumanity. And ASI shouldn't be built *right now* only because capabilities research is ahead of safety research. Once we know how to create an ASI that brings about a posthuman paradise, then pedal to the metal. As MIRI wrote in 2024: "We remain committed to the idea that failing to build smarter-than-human systems someday would be tragic and would squander a great deal of potential. We want humanity to build those systems, but only once we know how to do so safely." "Safely" means "it will bring about a posthuman paradise," and "potential" is a reference to our "glorious transhumanist future."
The problem is that this posthuman paradise would almost certainly result in the extinction of our species, which I oppose. And Yudkowsky has repeatedly said that's okay: "It’s not that I’m concerned about being replaced by a better organism, I’m concerned that the organism wouldn’t be better" -- Yudkowsky. See also this comment, where Yudkowsky talks about ending the human species once posthumanity arrives:
Yes I'm aware and your reporting on that has been informative and useful. It has lowered my opinion of many figures in AI Safety.
But I see the EA goal to put us in computers as more of a long term thing that will be done through manufacturing consent with an ASI. That is something that can be fought through public opinion, religion, journalism, different AIs, etc. They have a lot of legwork to do before normal people start uploading their minds into happiness computers. I also doubt it will be truly universal in the event they succeed, even Bostrom leaves a place for John the Savage as he does his virtual wireheading in Deep Utopia iirc. I will never upload my mind out of personal spite lol
We don't even get a chance to worry about that if we all get killed in 2 years by ASI though, preventing getting killed right now is the most important thing. So Yudkowsky meeting with Bernie and Tyson, doomers in the AI Doc, etc are a massive positive in terms of advancing the discourse on AI risk. Presenting a unified front against the accelerationists to prevent imminent death is the most important thing for the moment. Fight the (extremely important) philosophical battle about humanity after that has passed.
I take your points here. I'd push back on the idea that "the EA goal to put us in computers as more of a long term thing." If they built a "value-aligned" ASI next year, I think the first thing they'd do is use it to radically enhance themselves. In my book, I write about why this would have catastrophic consequences for humanity -- a link is below (unedited excerpt from the book that might be of interest!!). That makes me hesitant to join hands with the TESCREAL doomers. The way I'd describe the options that TESCREAL AI safety people are offering us: either human extinction (annihilation) or, alternatively, human extinction (posthuman utopia). Since I oppose human extinction, my fight is with both the AI accelerationists, AI companies, Altman, etc. and the TESCREAL doomers -- since both groups are advocating for a future in which our species dies out. Does that make sense? You could respond: yes, but the difference is annihilation is extinction-soon, while posthuman utopia is extinction-later, and the latter is better than the former! But I feel deeply uncomfortable locking arms with anyone who's okay with relative near-term extinction. If you get a chance to read the book excerpt, I'd love to know what you think. Please excuse any typos, etc.
I think the number 1 thing here is that the future is just going to suck if you're not a techno-utilitarian and there's no literal deus ex machina. We just have to acknowledge that and move forward even though it's intensely depressing.
I think your concerns are valid, but they rest on a lot of assumptions. They assume a sort of total institutional capture by the EAs (plausible) and a specific (although alarmingly plausible) philosophical bend that explicitly goes against what even the most sociopathic EA person (Bostrom) has talked about. By contrast, Scott Alexander (perhaps unrealistically) proposes a Nozickian Archipelago for the normies to live while the cyborgs and digital people go to space. Alexander is the god king of that space right now, so his word on that is pretty influential.
Ord and MacAskill both want to get rid of humans eventually, but they have the "Moral Uncertainty, Long Reflection" framework. We could feasibly win that debate. Like the EAs have a whole thing on "post-AGI governance" and they are clearly waiting a least a little before the big computer phase of things. Your doom scenario assumes both a change in their stated goals and the total collapse/capture of institutions. I think those are both alarmingly possible, but not guaranteed. Unaligned ASI extinction is guaranteed, so preventing that is the number 1 thing.
Also, Yud specifically wants a very long shutdown of AI development. If he were to somehow convince people to implement that, unlikely, it would buy a lot of time to debate the posthuman stuff. All of this sucks, but it sucks less than getting killed by Claude in 2030.
The accelerationists are literally trying to kill us off right now, no uncertainty. The hardcore versions of e/acc are literal cartoon villains! There is not really a big pro-human movement arguing for an AI shutdown because the centrists are captured by anti-luddism and the fringes are both obsessed with Israel/Epstein and dismiss AI capabilities. So right now all we have is the AI Safety people preventing us from getting killed. The way forward is to work with the AI Safety people to get people to take AI seriously, while also lowkey informing the normies of the end goals of the AI Safety people. I thought Taylor Lorenz's video (working with you) was largely a pretty good example of this, but it could've been more urgent about AI doom (using AI Safety arguments) right now.
I also think a lot of these people have differing goals. Like Altman is more of an "AI merge hivemind" guy, so are Diamandis and Dean Ball (who also says some part of the population will be able to stay biological, yippee). Andreessen and Thiel just want to be cyborgs rather than digital, etc. There's splits within SV as to what the end goal is, which undermines the idea of a unified posthuman monolith putting us in computers.
How does Ord and MacAskill actually imagine 8+ billion people -- Muslims, Christians, socialists, neo-Nazis, etc. -- all sitting down at a table to figure out what "our values" should be? Lol.
The rest of what you say is incredibly interesting and insightful. Thanks for sharing -- really appreciate it!
I’m quite serious about the gift subscription. This is an area where AI could truly benefit you. If utilized correctly, I promise this is the best option hands down.
Emile, I sometimes find myself disagreeing vehemently with your work, but as a fellow academic I am sympathetic to your publishing woes. The 13-month latency period in traditional publishing is not a function of quality control; it is a legacy artifact of human-intensive bottlenecks and asynchronous manual workflows.
With the topic you’ve chosen, information decay is a real risk, and waiting over a year to release findings can render data obsolete. Integrating AI-driven production suites in conjunction with a frontier level consumer LLM allows you to bypass these administrative hurdles by automating the "grunt work" of publication in a matter of days.
I know that’s not what you want to hear. This isn't about replacing the authorial voice, but about removing the 20th-century friction between a finished manuscript and a global audience.
By utilizing AI (I would recommend either Grok or an abliterated locally run model like Heretic) as a technical multiplier, you retain full authorial autonomy while collapsing the production timeline.
Current AI architectures are eminently capable of maintaining structural consistency across thousands of pages, ensuring that citations, cross-references, and technical nomenclature remain precise without the need for multiple rounds of manual back-and-forth with a traditional editor.
Think of your upcoming book not as a static object waiting for a slot in a publisher's schedule, but as a high-speed telemetry stream that can be deployed to market the moment the research is finalized. For a scientist, the objective is the efficient transfer of information; AI is simply the most optimized protocol for that transfer.
If you are interested I’ll purchase you a gift subscription to Super Grok. You will be surprised and pleased by what it is capable of.
Woah, this article basically covers almost every thinking I have been doing about AGI. I agree with a lot of what you say. The amateur philosopher (the one who didn't read up in detail about Yudkowski, nor on the difference between AGI and ASI) in me has a few things to share.
Unless Yudkowski has somehow mathematically proven there is some kind of universal morality that supersedes anything human-wrought, I think his views as you describe them might contradict themselves. If we talk about "greater good", shouldn't it be always about the greater good for mankind? How does eliminating mankind achieve that? Whichever way you look at it, this is all thought out by and for humans and I doubt whether we can know what is good for "the rest of the universe". That'd be quite arrogant. In short, eliminating mankind would also wipe out all our values and whatever is left would literally have no meaning at all. Well that was just the logical way of looking at it. The human in me, the one that is consciously aware he is subject to all kinds of human mechanism (emotion, social values, etc), 100% agrees about the atrociousness of his view.
As for AGI, again I agree with your estimate where we likely stand. And also with the dangers it poses. But I fear that the Djinn that is "the road to AGI" is out of the bottle. And I cannot conceive of a way to put it back. I can't help but think that the scenarios in which we might be able to put it back, may be too bleak to consider, even though pure logic dictate we should (compare how we deal with climate change). In that respect the future really looks bleak. Things are even bleaker when we look at the number of challenges we currently face, that shouldn't have been existential, but still are due to human stupidity: neglecting vulnerabilities of OpenClaw like architectures, sloppy software engineering practices in the very company that claims to be on the forefront of alignment science (Anthropic), government systems of powerful countries that don't have guardrails against lunatics with genocidal tendencies, BigTech that lie there asses off in the name of profit, climate change, and the list goes on. All problems that are theoretically much easier to solve than the existential problems caused by AGI/ASI.
Despite all of the above, I entertain a view of which I am still pondering whether it is naive (it probably is). AGI is dangerous for exactly the reasons you name. The alignment problem is mathematically unsolvable. The Djinn cannot be put back. So where does that put us? Well, imho, with the only skill humanity has time and time again shown that they can perform near miracles with: Engineering. However small the chances that Engineering will save us, it is the only option we have, so we MUST take it. Engineering what? Well a good enough solution to the alignment problem: heuristics, patches, guardrails, etc etc and probably a huge amount of luck. Again, it is not much to go on, but it is all we have. So we better get going and not only pray for, but build that miracle.
Congrats on the book... A year is a very long turnaround time maybe you could split the language editions or keep the rights for English and self publish but its very long...
projectallende@gmail.com
Here's a thing I don't understand about your responses to Yudkowski, specifically, and others more generally. When I listen to the interviews in which Yudkowski says something like, "If the only way to get godlike beings into the universe is to sacrifice all human beings, then I'd ultimately be willing to make that trade," it's never in wistful tones. It's generally in mild exasperation: somebody is presenting him with the start of a slippery slope argument, and he sidesteps the whole thing by saying, "Sure, at the bottom of the slide, if there's only A or only B, I choose A. Now, can we talk about something that might actually happen? Something that might actually matter?"
It's not that all of Yudkowski's positions are mild; "mild" isn't really his brand. There's any number of things he says that you might disagree with, and a variety of ways you could make that disagreement. But this recurring bit of yours makes you difficult for me to take seriously.
"this recurring bit of yours makes you difficult for me to take seriously." Hmm.
Are you joking? I don't take anyone who uses omnicidal language seriously, nor do I take anyone who's okay with omnicidal langauge seriously. Yudkowsky has repeatedly expressed pro-extinctionist sentiments. I've catalogued these. They are unacceptable. They are, frankly, *fucking insane*. It is *insane* to suggest that humans should be cleared off Earth to leave it as a nature reserve, and that it may be unethical to have biological kids once the posthuman era arrives. Those are some of the most atrocious things I've ever heard anyone say, ever. If you don't agree, then frankly I'm a little frightened by you.
I've read through your documented evidence of the atrocious things Yudkowski and others say. I've followed your links to the original sources. And I often find that your reading of their statements scans like somebody made a bald transcript of words spoken, shorn of tone and body language, then you read it as if by a character actor whose job was to be an over-the-top villain.
I would agree that somebody who scanned the world from a balcony and said, grimly, "These humans. These petty fucks. Who needs 'em? Not I. Let's crank up the robot overlords, wipe out these scum, and get on with the god minds!" is an evil bastard we should do something about.
But that's never the way the people in your links are talking. They are parsing thought experiments, addressing slippery slopes, staking out boundary conditions, etc. There's not a sense I have that Yudkowski has a "kill all the humans" button on his desk and he's slapping it over and over with a lunatic grin.
"I would agree that somebody who scanned the world from a balcony and said, grimly, "These humans. These petty fucks. Who needs 'em? Not I. Let's crank up the robot overlords, wipe out these scum, and get on with the god minds!" is an evil bastard we should do something about."
This is very close to what Yudkowsky said on Bankless. Once posthumanity arrives, it may be (he says using offensive language) unethical to keep humanity around. He leaves open how exactly we die out -- maybe through forced infertility rather than outright physical violence. But his vision of "utopia" doesn't include us. It doesn't include me or my family. If you're okay with that, fine.
Well, that was disappointing: Yudkowsky (I've been misspelling his name) didn't talk about posthumanity on Bankless, the one time I could find him there in Feb 2023, and he only ever said that AI would kill everyone as if killing everyone was bad.
It's at the very end, not part of the main episode (though you can find the recording on Twitter/X). Yudkowsky says:
------------
Eliezer: For one thing, I predict that if there's a glorious transhumanist future (as it is sometimes conventionally known) at the end of this, I don't predict it was there by getting like, “coexistence” with superintelligence. That's, like, some kind of weird, inappropriate analogy based off of humans and cows or something.
I predict alignment was solved. I predict that if the humans are alive at all, that the superintelligences are being quite nice to them.
I have basic moral questions about whether it's ethical for humans to have human children, if having transhuman children is an option instead. Like, these humans running around? Are they, like, the current humans who wanted eternal youth but, like, not the brain upgrades? Because I do see the case for letting an existing person choose "No, I just want eternal youth and no brain upgrades, thank you." But then if you're deliberately having the equivalent of a very crippled child when you could just as easily have a not crippled child.
Like, should humans in their present form be around together? Are we, like, kind of too sad in some ways? I have friends, to be clear, who disagree with me so much about this point. (laughs) But yeah, I'd say that the happy future looks like beings of light having lots of fun in a nicely connected computing fabric powered by the Sun, if we haven't taken the sun apart yet. Maybe there's enough real sentiment in people that you just, like, clear all the humans off the Earth and leave the entire place as a park. And even, like, maintain the Sun, so that the Earth is still a park even after the Sun would have ordinarily swollen up or dimmed down.
Yeah, like... That was always the things to be fought for. That was always the point, from the perspective of everyone who's been in this for a long time. Maybe not literally everyone, but like, the whole old crew.
Ryan: That is a good way to end it: with some hope. Eliezer, thanks for joining the crypto community on this collectibles call and for this follow-up Q&A. We really appreciate it.
michaelwong.eth: Yes, thank you, Eliezer.
Eliezer: Thanks for having me.
While I technically have an X.com account, I only use it when the only source to something is a link. Perhaps I can task an AI with finding the link. I think this transcript shows what I'm talking about, though.
In several conversations I listed to, Yudkowsky has a couple of registers for talking about outcomes. The further out, the more hypothetical, the less nuanced positions become: it becomes a case of "would you rather" with some extreme options on the table, and the person gets to examine their core values against it. In one, he began dispirited, and he shrugged his way through saying something like, "If there's not something that cares about having ice cream, then I don't want it," which was about trying to draw distinctions between an emotional range (not necessarily a modern human emotional range) and being an advanced database that has access to all the information. Pointing at a condition of qualia, perhaps.
By the time we're talking about Dyson spheres and whatnot, we're in an alien situation. Imagine an ancient Greek being asked to opine about a far-future democracy: "Would you rather Greek society continue on as a dictatorship, or a non-Greek society continuing on as a democracy?" In neither case would the life of those future people be quintessentially Greek as that person knows themselves to be. So it's about parsing, "better my descendants in hell," or "better none of my descendants, but basking in the demos of my people"?
To then say that the ancient Greek who prefers their descendants not in hell is a "pro-extinctionist for Greeks" would be unfair. (Or that the ancient Greek prefers their descendants alive in hell is a self-hating sadist.)
[I have thoughts on the ethics of having children of different traits under different contexts, but seeing your newest post, perhaps that's the better context for them.]
Note that when Yudkowsky talks about "humanity," he's typically using the Broad Definition to mean "our species and our posthuman descendants." On that definition, our species could die out literally next year without "human extinction" having happened.
Yudkowsky is a transhumanist. He's on Team Posthuman, not Team Human, and hence objects to ASI *only insofar* as it might prevent us from realizing the "glorious transhumanist future."
Essentially agree with this. Parts of the AI Safety movement are not actually that pro-human and it's good that Emile points that out, but they assume the worst possible interpretation of every statement. Big problems, like preventing the imminent extinction of humanity, require big coalitions.
I would not want Yudkowsky (or frankly literally anyone in AI) in charge of the world, but Yud is much closer to the correct position than someone trying to get ASI into the world to maximize entropy. Or a normie businessman getting conned by said person.
The accelerationists have normie business people, machine god worshippers, and human extinctionists under the same tent. Fighting over hypothetical sacrifices is something to be managed later.
I also see no reason to think "Nowhere close to AGI" is a particularly likely position. I think we're a decade away at most, assuming no world war or economic collapse. We might even be a year away, which would be.... not great
"like preventing the imminent extinction of humanity" --> but, to be clear, for TESCREALists this means "humanity" on the Broad Definition, or "extinction" in the sense of final extinction. That is not, at all, what I and the large majority of people on Earth mean when we talk about "human extinction." I actually care about avoiding the extinction of our species (terminal extinction, on the Narrow Definition). People like Yudkowsky, and most others in the TESCREAL movement, don't. https://link.springer.com/article/10.1007/s10790-025-10072-7
Perhaps it's my fault for not explaining this well enough. Most TESCREALists who say they care about "avoiding human extinction" don't mean what nearly everyone else means. Their definitions of "human" and "extinction" are different and idiosyncratic. They are not on Team Human -- to the contrary, even folks like Yudkowsky explicitly advocate for a view in which we could go (terminally) extinct in the near future. I find that atrocious, but you're welcome to disagree. If so, I don't know why you'd be interested in my Team Human newsletter articles!
I pay a subscription to you because I disagree with you. I think your work is valuable, and I appreciated the paper you reference here when you first published it.
I think you are wrong when you magnify one statement from a thought experiment or hypothetical as if it were the speaker's most important personal creed.
It's not just one statement, though! It's many. It's also built-into his transhumanism. The ultimate goal is to create a "utopian" world of digital space brains. The survival of our species, humanity, matters only insofar as it's necessary to create/become posthumanity. And ASI shouldn't be built *right now* only because capabilities research is ahead of safety research. Once we know how to create an ASI that brings about a posthuman paradise, then pedal to the metal. As MIRI wrote in 2024: "We remain committed to the idea that failing to build smarter-than-human systems someday would be tragic and would squander a great deal of potential. We want humanity to build those systems, but only once we know how to do so safely." "Safely" means "it will bring about a posthuman paradise," and "potential" is a reference to our "glorious transhumanist future."
The problem is that this posthuman paradise would almost certainly result in the extinction of our species, which I oppose. And Yudkowsky has repeatedly said that's okay: "It’s not that I’m concerned about being replaced by a better organism, I’m concerned that the organism wouldn’t be better" -- Yudkowsky. See also this comment, where Yudkowsky talks about ending the human species once posthumanity arrives:
https://www.realtimetechpocalypse.com/p/utter-madness/comment/240623527
Yes I'm aware and your reporting on that has been informative and useful. It has lowered my opinion of many figures in AI Safety.
But I see the EA goal to put us in computers as more of a long term thing that will be done through manufacturing consent with an ASI. That is something that can be fought through public opinion, religion, journalism, different AIs, etc. They have a lot of legwork to do before normal people start uploading their minds into happiness computers. I also doubt it will be truly universal in the event they succeed, even Bostrom leaves a place for John the Savage as he does his virtual wireheading in Deep Utopia iirc. I will never upload my mind out of personal spite lol
We don't even get a chance to worry about that if we all get killed in 2 years by ASI though, preventing getting killed right now is the most important thing. So Yudkowsky meeting with Bernie and Tyson, doomers in the AI Doc, etc are a massive positive in terms of advancing the discourse on AI risk. Presenting a unified front against the accelerationists to prevent imminent death is the most important thing for the moment. Fight the (extremely important) philosophical battle about humanity after that has passed.
I take your points here. I'd push back on the idea that "the EA goal to put us in computers as more of a long term thing." If they built a "value-aligned" ASI next year, I think the first thing they'd do is use it to radically enhance themselves. In my book, I write about why this would have catastrophic consequences for humanity -- a link is below (unedited excerpt from the book that might be of interest!!). That makes me hesitant to join hands with the TESCREAL doomers. The way I'd describe the options that TESCREAL AI safety people are offering us: either human extinction (annihilation) or, alternatively, human extinction (posthuman utopia). Since I oppose human extinction, my fight is with both the AI accelerationists, AI companies, Altman, etc. and the TESCREAL doomers -- since both groups are advocating for a future in which our species dies out. Does that make sense? You could respond: yes, but the difference is annihilation is extinction-soon, while posthuman utopia is extinction-later, and the latter is better than the former! But I feel deeply uncomfortable locking arms with anyone who's okay with relative near-term extinction. If you get a chance to read the book excerpt, I'd love to know what you think. Please excuse any typos, etc.
https://www.realtimetechpocalypse.com/p/20a111bb-ea1d-4ec0-8293-92837c528a5a?postPreview=paid&updated=2026-04-09T11%3A14%3A18.584Z&audience=everyone&free_preview=false&freemail=true
I think the number 1 thing here is that the future is just going to suck if you're not a techno-utilitarian and there's no literal deus ex machina. We just have to acknowledge that and move forward even though it's intensely depressing.
I think your concerns are valid, but they rest on a lot of assumptions. They assume a sort of total institutional capture by the EAs (plausible) and a specific (although alarmingly plausible) philosophical bend that explicitly goes against what even the most sociopathic EA person (Bostrom) has talked about. By contrast, Scott Alexander (perhaps unrealistically) proposes a Nozickian Archipelago for the normies to live while the cyborgs and digital people go to space. Alexander is the god king of that space right now, so his word on that is pretty influential.
Ord and MacAskill both want to get rid of humans eventually, but they have the "Moral Uncertainty, Long Reflection" framework. We could feasibly win that debate. Like the EAs have a whole thing on "post-AGI governance" and they are clearly waiting a least a little before the big computer phase of things. Your doom scenario assumes both a change in their stated goals and the total collapse/capture of institutions. I think those are both alarmingly possible, but not guaranteed. Unaligned ASI extinction is guaranteed, so preventing that is the number 1 thing.
Also, Yud specifically wants a very long shutdown of AI development. If he were to somehow convince people to implement that, unlikely, it would buy a lot of time to debate the posthuman stuff. All of this sucks, but it sucks less than getting killed by Claude in 2030.
The accelerationists are literally trying to kill us off right now, no uncertainty. The hardcore versions of e/acc are literal cartoon villains! There is not really a big pro-human movement arguing for an AI shutdown because the centrists are captured by anti-luddism and the fringes are both obsessed with Israel/Epstein and dismiss AI capabilities. So right now all we have is the AI Safety people preventing us from getting killed. The way forward is to work with the AI Safety people to get people to take AI seriously, while also lowkey informing the normies of the end goals of the AI Safety people. I thought Taylor Lorenz's video (working with you) was largely a pretty good example of this, but it could've been more urgent about AI doom (using AI Safety arguments) right now.
I also think a lot of these people have differing goals. Like Altman is more of an "AI merge hivemind" guy, so are Diamandis and Dean Ball (who also says some part of the population will be able to stay biological, yippee). Andreessen and Thiel just want to be cyborgs rather than digital, etc. There's splits within SV as to what the end goal is, which undermines the idea of a unified posthuman monolith putting us in computers.
New article is relevant: https://www.realtimetechpocalypse.com/p/if-anyone-becomes-posthuman-everyone. What do you think of it? The "long reflection" idea is a nonstarter for me. It's like saying, "I'll tell you what, the best thing to do right now is establish world peace." Like, um, okay?
How does Ord and MacAskill actually imagine 8+ billion people -- Muslims, Christians, socialists, neo-Nazis, etc. -- all sitting down at a table to figure out what "our values" should be? Lol.
The rest of what you say is incredibly interesting and insightful. Thanks for sharing -- really appreciate it!
I’m quite serious about the gift subscription. This is an area where AI could truly benefit you. If utilized correctly, I promise this is the best option hands down.
What gift subscription? Sorry for not following!
For super grok to help with publishing the book.
Emile, I sometimes find myself disagreeing vehemently with your work, but as a fellow academic I am sympathetic to your publishing woes. The 13-month latency period in traditional publishing is not a function of quality control; it is a legacy artifact of human-intensive bottlenecks and asynchronous manual workflows.
With the topic you’ve chosen, information decay is a real risk, and waiting over a year to release findings can render data obsolete. Integrating AI-driven production suites in conjunction with a frontier level consumer LLM allows you to bypass these administrative hurdles by automating the "grunt work" of publication in a matter of days.
I know that’s not what you want to hear. This isn't about replacing the authorial voice, but about removing the 20th-century friction between a finished manuscript and a global audience.
By utilizing AI (I would recommend either Grok or an abliterated locally run model like Heretic) as a technical multiplier, you retain full authorial autonomy while collapsing the production timeline.
Current AI architectures are eminently capable of maintaining structural consistency across thousands of pages, ensuring that citations, cross-references, and technical nomenclature remain precise without the need for multiple rounds of manual back-and-forth with a traditional editor.
Think of your upcoming book not as a static object waiting for a slot in a publisher's schedule, but as a high-speed telemetry stream that can be deployed to market the moment the research is finalized. For a scientist, the objective is the efficient transfer of information; AI is simply the most optimized protocol for that transfer.
If you are interested I’ll purchase you a gift subscription to Super Grok. You will be surprised and pleased by what it is capable of.