What Is AI Successionism?
The term “AI successionism” has been making the rounds recently, but I find it dangerously misleading. For example, the statistician Olle Häggström wrote a Substack article about it not long ago,1 as did Sigal Samuel for Vox. The latter defines AI successionists as those
who think that artificial intelligence is our rightful heir — the next step in cosmic evolution. Since they believe AIs could become our moral superiors, they argue it’s actually wrong to try to keep the machines down, or even to align them with human values, as most AI companies aim to do. Instead, we should usher in artificial intelligence as a successor to humanity and hand over the world to it. Even if that means we go extinct.2
The problem is that “AI successionism” conceals the dangerous fact that the TESCREAL movement as a whole is intimately bound up with a pro-extinctionist outlook, and that many TESCREAL pro-extinctionists reject AI successionism. What does this mean?
Most TESCREALists concur — some implicitly, but many explicitly — that posthumanity should usurp our biological species, perhaps in the very near future. However, they want these posthumans to be extensions of us — transformed versions of human beings.
The goal is to build an “aligned” ASI that enables them to merge with machines, upload their minds to computers, and join the ASI as a digital super-being flitting about the universe, literally owning galaxies while living in giant computer simulations powered by Dyson swarms.
This contrasts with the AI-successionist view that we should build, de novo, autonomous ASIs that simply displace us like an invasive species let loose on an island. In one case, our evolutionary lineage is radically transformed through “radical enhancement” and digitization, while, in the other, digital beings that we create but are not otherwise evolutionarily related to us take our place.
This is a major internecine dispute within the pro-extinctionist community, and I suspect that there will be increasingly public debates between TESCREAL extremists not about whether pro-extinctionism is good, but about which type of pro-extinctionism is best.
The danger is that, by focusing entirely on AI successionism, readers may get the dangerously false impression that it’s the only form of pro-extinctionism in the Valley. It is not. Many pro-extinctionists strongly object to the particular form of pro-extinctionism called “AI successionism.”
Some Examples
Sam Altman doesn’t appear to be an AI successionist. Yet he argues that the only way “we” will “survive” the advent of ASI is to “merge” with machines. He’s already taken steps to become software himself, just like the ASI, by signing up with a company called Nectome to have his brain digitized if he dies before the Singularity (assuming the Singularity “goes well,” meaning that it results in a cosmic posthuman utopia).
Similarly, Dario Amodei writes that
if we do get a century of medical progress in a few years, it is possible that we will greatly increase the human lifespan, and there is a chance we also gain radical capabilities like the ability to increase human intelligence or radically modify human biology.
This is a reference to the possibility of becoming posthuman through what transhumanists like Nick Bostrom call “extreme human enhancements,” i.e., enhancements that confer at least one “posthuman capacity.”
Amodei adds that “there is also the issue of ‘uploads’ or ‘whole brain emulation,’ digital human minds instantiated in software, which might someday help humanity transcend its physical limitations,” though he says this carries risks. (That is, risks we’ll need to overcome, because transcendence is the ultimate end-goal.)
Here I would like to point out that Amodei would probably never publicly say that he thinks our biological species should die out once posthumanity arrives. But reflect on the likely outcome of posthumanity rising up to rule the world. On the one hand, our species would become irrelevant and obsolete in every way: technologically, scientifically, economically, and politically. At least that’s the goal — for “superior” posthumans to run the show. On the other hand, we would continue to suck up valuable finite resources that these “superior” posthumans with “Jupiter-sized minds” (quoting Bostrom) could use for “better” purposes.
In other words, we would become not just useless, but impediments. We would serve no purpose while taking up space and wasting resources. Why, then, would anyone expect us to persist for more than a brief moment once the posthuman era commences?
To put this differently, Silicon Valley pro-extinctionism advances a two-step agenda. First, create or become a new posthuman species. Second, replace humanity with this new posthuman species. One can theoretically accept the first while rejecting the second. Those who do this advocate for what I call the “coexistence view,” in contrast to the “replacement view” championed by outright pro-extinctionists.
However, I argue in detail here that realizing the first step while avoiding the second step — as coexistence view advocates say they want — is totally implausible. Why on Earth would posthumanity keep us around if we’re useless and in the way? Obviously, our species would die out in a world ruled and run by posthumans!
Even more, people like Eliezer Yudkowsky argue that there may be ethical reasons to get rid of us. As he once said, “are we, like, kind of too sad in some ways” to keep around? He claims (like most other TESCREALists) that the ideal situation would be one in which digital beings live in huge computer simulations powered by Dyson swarms, and likens the continuation of biological humans in a posthuman world to someone today intentionally choosing to have a disabled child (though his language is even more offensive and ableist than this).
These are just a few examples of non-AI successionist pro-extinctionists. See this comprehensive article of mine for lots more.
I Am Begging Journalists to Wake Up
To reiterate my point: the people above oppose AI successionism. Yet they hold views that are either explicitly (Altman, Yudkowsky) or implicitly (Amodei) pro-extinctionist.
It is absolutely paramount for people to understand this. The AI succesionists aren’t the only faction in Silicon Valley that advocates for a future in which our species is systematically sidelined, marginalized, disempowered, and ultimately eliminated.
To the contrary, a very large percentage of the most powerful people in the ASI race embrace a digital eschatology according to which humanity will and should be eventually usurped by posthumanity — in this case, understood as a kind of evolutionary extension of our species, digitized and living on the cloud.
I strongly recommend that whenever journalists — or anyone — bring up AI successionism, they also mention that this is a vocal minority within a much broader pro-extinctionist movement.
The most relevant and important distinction is between those who are on Team Human and those who are on Team Posthuman. All the pro-extinctionists — from Yudkowsky to Verdon, Richard Sutton to Toby Ord — are on Team Posthuman. They are a threat to our species. Their eschatological visions pose dire risks to the survival of humanity. They are the enemy, a word I don’t think is strong enough.
But what do you think? What might I be wrong about? As always:
Thanks for reading and I’ll see you on the other side!
In which he notes that Nick Bostrom is a bit of an AI successionist. Whether that particular claim is true or not, Bostrom pretty clearly holds a pro-extinctionist view — by which I mean he would likely endorse terminal extinction without final extinction. (In other words, our species’ survival matters only insofar as it’s necessary to become or create our (worthy) posthuman successors, after which we should kick the bucket.)
This is not clearly stated, though. An uploaded human mind would be AI, but that’s not what Samuel means. Rather, she’s pointing to the idea that autonomous ASIs, which are not direct extensions of individual humans, inherit the world.