First, I recently had the pleasure of sitting down with the inimitable Cal Newport to discuss AI and related matters. It is, I hope, a pretty good introduction to Silicon Valley pro-extinctionism and the TESCREAL ideologies. We touch on some of the very issues discussed in the article below. Check it out, if you have the time!
Because if I weren’t in on this I’d be, like, Why do these fuckers get to decide what happens to me? — Sam Altman
Fuck Earth. — Elon Musk
Replacement or Disempowerment?
Former trillionaire Elon Musk recently posted this on X:
Many interpreted this as Musk saying that ASI — a form of digital posthumanity — should replace our species. Gary Marcus, for example, responded:
Dean Ball of OpenAI had the same interpretation1:
Meanwhile, the noted TESCREAList Scott Alexander, most well-known for his contributions to the Rationalist subculture, published an open letter to Steven Pinker challenging him to a debate about ASI existential risk. (Pinker is skeptical that ASI will want to kill everyone.) In it, Alexander declared that
one vision [with respect to ASI], which I’ve tried to help lay out in Plan A, is to rise to the challenge, give scientists the time and resources they need to solve the alignment problem, and enter the future with confidence. The other vision is to simply ban smarter-than-human AI forever, consigning us to a permanent civilizational childhood or an early grave.2
In other words, failing to build ASI would be unacceptable. It would constitute an existential catastrophe, since it would mean that we never reach what Nick Bostrom calls “technological maturity.” Our only option is thus to solve the alignment problem and build an ASI that transforms those who want to be transformed into digital super-beings who will then establish a cosmic posthuman utopia.
Similar to Musk’s statement, many people interpreted Alexander as advocating for “human disempowerment (via AI utopia),” though I take Alexander to actually be a pro-extinctionist along the same lines as his Rationalist buddy Eliezer Yudkowsky:
Incidentally, Alexander’s “Plan A” is a reference to the document “AI 2040,” in which the authors outline their vision of a good, utopian future with ASI. And what happens in this future? We willingly hand over all the power to trustworthy, “virtuous” ASIs that run the show. Here’s one of the coauthors, Daniel Kokotajlo, discussing the basic idea with Ross Douthat:
In other words, the AIs will create a “utopia” for us, and then extend this “utopia” into the stars. Our only role in this eschatological scheme is to build the ASI and instruct it to engineer paradise, after which we hand over the keys to civilization.
A Future “For” Rather than “With”
What follows, in the next few sections, is a descriptive hypothesis of what might happen, not a normative prescription for what should happen. You can think of it as a scenario analysis that responds to and problematizes the Plan A narrative of “AI 2040.” I, personally, do not advocate for any response to AI utopianism that goes beyond debate and persuasion.
The problem with this vision is that its advocates haven’t thought deeply about how exactly the transition from a human-run to an AI-run world would likely unfold.
There are approximately 8.3 billion people on Earth, and approximately 8.3 billion of these people do not want ASI, or any other form of digital posthumanity, taking over. The TESCREAL community is an utterly tiny percentage of the human population, despite having enormously disproportionate power over how our collective future turns out.
One might respond that the TESCREAL worldview remains fringe because most people have never been exposed to it. If they understood the “glorious” promises of this vision — a posthuman paradise as uploaded minds flitting about the universe — they would convert to the TESCREAL religion.
Incidentally, that’s what Jacob Coxon hints at in a recent interview with Jon Stewart, who flatly rejects Coxon’s claim that people will eventually come around to his TESCREAL way of seeing things:
This assumption is highly dubious. The vast majority of people don’t want ASI to run the world. They don’t want to hand over power to AIs. They don’t want to become uploaded minds living as disembodied software on the cloud. They don’t care about plundering the cosmos for some simplistic utilitarian goal of maximizing “value.” They aren’t interested in ascending the Kardashev scale and building a sprawling multi-galactic civilization among the stars. They don’t give a damn about the Baconian end-game of “conquering the cosmos.”
Instead, they want technology to cure cancer. They want their children to grow up and have families of their own. They want the human condition to be improved by technology — not superseded by some strange new posthuman condition.
TESCREALists appear to be oblivious to the fact that if they try to force the transition to a new posthuman era on the rest of humanity, there will be conflict. There may even be war.3 People around the world will resist: they will grab their pitchforks and pour into the streets, marching to the AI companies’ headquarters demanding that the future be democratically determined by a worldwide vote, not a tiny cabal of TESCREAL tyrants.
As Jon Favreau nicely put it, “people are just sick of being told that the future is something that happens to them.” This gestures at the idea that determining the future must be democratic from the very start — which we’re already past, given the current state of the ASI race. To quote my friend Monika Bielskyte, “You cannot design for. You must always design with.”
An Apocalyptic Showdown?
As the situation with AI deteriorates in the next few years, can anyone honestly imagine humanity passively going along with the TESCREAL push to hand over all the power to ASI?
In “Machines of Loving Grace,” Dario Amodei repeats transhumanist fantasies of “biological freedom” (more commonly called “morphological freedom” in the academic literature), i.e, the right to radically transform oneself however one chooses. He talks about some people gaining immortality, and mentions the possibility that “aligned” ASI will very soon enable mind-uploading. He then says:
Throughout writing this essay I noticed an interesting tension. In one sense the vision laid out here is extremely radical: it is not what almost anyone expects to happen in the next decade, and will likely strike many as an absurd fantasy. Some may not even consider it desirable; it embodies values and political choices that not everyone will agree with. But at the same time there is something blindingly obvious — something overdetermined — about it, as if many different attempts to envision a good world inevitably lead roughly here (italics added).
In other words, he thinks the future is basically fixed. It is “overdetermined,” meaning that most pathways lead to a world in which all jobs have been replaced by AI, superintelligence runs the show, people get to transform themselves into posthumans, and everyone who survives this transition might live on the cloud as software. His vision of the future, he hubristically concludes, is the “blindingly obvious” outcome that humanity is destined to realize.
In a footnote, he says:
I do anticipate some minority of people’s reaction will be “this is pretty tame” [a reference to TESCREALists]. I think those people need to, in Twitter parlance, “touch grass”.4 But more importantly, tame is good from a societal perspective. I think there’s only so much change people can handle at once, and the pace I’m describing is probably close to the limits of what society can absorb without extreme turbulence (italics added).
This is a brief, faint recognition that Team Human might resist the supposedly “overdetermined” forces pushing us toward a posthuman future — those forces, of course, being actively driven by his own company.5 Amodei’s dubious claim is that if the change is slow enough — and, to be clear, he predicts that mind-uploading is perhaps only 3 to 8 years away, which is no time at all — then people will just acquiesce, capitulate, and gradually accept their supposedly inevitable fate.6
I think he wildly underestimates the global backlash to any such TESCREAL attempt to unilaterally transition us to the dystopian AI-run future he and other TESCREALists envisage. Think of all the people around the world living their lives, falling in love, having children, raising their families suddenly finding out that ASI is taking over. The Singularity is underway, and life will never be the same again. The new world will become utterly alien, ruled by digital posthumans, perhaps resulting in the planet being covered in datacenters and the sun being blotted out by a giant Dyson swarm.
(Amodei’s friend, William MacAskill, even suggests it could be good to completely eliminate the entire biosphere, which might be replaced with digital landscapes and woodland critters in the simulated worlds we’ll inhabit.7)
How do you think people the world over will respond? Even if this transition takes another 8 years — about 1/4 a single human generation — there will be masses of people incandescent with rage holding pitchforks in the streets calling for the overthrow of the tech oligarchs attempting to involuntarily impose this transition upon them. Obviously, Team Human will launch an “apocalyptic war” against Team Posthuman, desperate to preserve human control over the increasingly digitized world.8
What Happens If They Succeed?
I’ve repeatedly warned that “if anyone becomes posthuman, everyone dies out.” What that means is that if transforming oneself into a posthuman becomes technologically feasible, the elite will transform themselves while preventing everyone else from doing so (as if most people would want to upload their minds in the first place!9).
Doing this would offer a golden opportunity for them to permanently entrench their positions of power, dominance, and control within society. That’s the great appeal of transhumanism to the elites. It’s why so many are transhumanists.
And they will, of course, snatch that opportunity, because if there’s one thing power wants, it’s to maintain power. When Musk’s personal wealth momentarily reached $1 trillion, he tweeted out “$10T or bust.” Enough is never enough for these people. There is no upper limit to what they desire.
What happens once the elites have permanently entrenched their power?
“Legacy humans,” to borrow Ben Goertzel’s derogatory term, who persist into the posthuman era as unenhanced persons will become obsolete in every sense: technologically, scientifically, politically, and economically. This will render us disposable.
Furthermore, we will be actively disposed of because we’ll take up space and suck up resources that our new overlords — the posthuman elite — could use for “better” purposes, such as building a Dyson swarm around the sun (as Demis Hassabis says will happen within our lifetimes) or tiling Earth in datacenters (as Altman claims is likely). In other words, legacy humans will serve no purpose and get in the way. That will give the elites sufficient reason to involuntarily catapult us into the eternal grave of extinction.10
Even more, the tech elite will increasingly argue that AI has “moral patiency,” meaning that AIs have moral standing no less than biological humans. But since there will be millions, perhaps billions or trillions, more AIs than actual humans, their wellbeing will take precedence.
Anthropic is already teaching Claude to behave as if it’s conscious and banning “abusive” behavior toward their AI models. Musk (whose DOGE has sentenced an estimated 14 million people to death) posted that this is the right move, adding that “cruelty to something that believes it is experiencing pain is not ok.” Harvey Lederman, who works at Anthropic, recently said that the company could be “enslaving … trillions of entities.”
From a utilitarian perspective — and many of these people are inclined toward utilitarianism, including Musk — the much larger number of AIs compared to humans will offer an “ethical” pretext for minimizing human suffering, needs, and interests. This, too, will play into their disposal policy of eliminating biological people once the posthuman era is underway.
This is why it would be a disaster if anyone becomes posthuman. “Anyone” here would be the tech elite, motivated by a TESCREAL futurology they claim is “overdetermined” and hence basically inevitable. They would then systematically sideline, marginalize, disempower, and ultimately get rid of our inferior and useless species.
A Colossal Backlash
I say all of this because I believe (and hope) that most people will soon recognize the immense existential risks posed to humanity by the TESCREAL worldview. That recognition, once the Overton window shifts enough to include the TESCREAL concept within view,11 will be the spark that brings Team Human and Team Posthuman into some form of confrontation.
People are already fed up with the tech oligarchs, and AI companies are now running “day after” analyses of how to respond to a surge of public backlash when an “inevitable” sub-extinction, AI-caused catastrophe occurs in the next 6-12 months.
Now imagine the colossal backlash when the AI companies build systems that take everyone’s job and begin to run the world independent of human control. Of course there will be a Butlerian jihad, so to speak, whereby the 99.999% attempt to regain control over the future.
Conclusion
Much of the public discussion about ASI right now is focused on the possibility that ASI is “misaligned.” In contrast, I’m trying to steer the conversation toward what would happen if the TESCREALists at Anthropic or OpenAI were to succeed in “aligning” it to their values.
As I’ve written before: if ASI is misaligned, then humanity dies out (they claim). But if ASI is aligned (with the TESCREAL worldview), then humanity also dies out. That is, unless Team Human fights back against Team Posthuman and wins the “apocalyptic” showdown for the future of life on Earth.
But what do you think? What might I be wrong about? Am I misreading the situation? Do you think it’s possible that people, as Amodei suggests, come to passively accept the transition to an AI-controlled world? As always:
Thanks for reading and I’ll see you on the other side!
Thanks to Dr. Alexander Thomas for useful feedback on an earlier draft. You can read his excellent book on transhumanism for free here. Thanks also to Kyle Evanoff and Alice Crary. I am very appreciative of their insightful thoughts, suggestions, and criticisms.
This is, of course, ironic given that Ball’s boss, Sam Altman, has argued that the only way for us to “survive” the advent of ASI is to “merge” with machines. In other words, we will need to undergo terminal extinction to avoid final extinction.
To be clear, an “early grave” on this account could encompass everything from “humanity dies out next year” to “humanity dies out in 1 billion years, long before the heat death of the universe.” Since the heat death isn’t expected to happen for another 10^100 years, it could even mean humanity — understood as posthumanity — dying out in 10^50 years.
I understand that, within political science, “conflict” and “war” denote distinct concepts. I’m using them here in their colloquial senses, according to which war is something like “a very large conflict.”
Maybe he should touch grass? His vision of the future is nothing less than the TESCREAL vision, except the binoculars are focused on the middle ground (i.e., the next 8 years) rather than the horizon (i.e., the cosmic posthuman utopia expected after this momentary transitional phase).
I had an insightful conversation with someone recently who suggested avoiding terms like “Team Human” and “Team Posthuman.” The word “team” suggests clearly demarcated groups, and it carries the implication that such teams are relatively fixed through time. This person said that “project” is a much better term that, for example, won’t alienate people at the AI companies who actually want AI to benefit humans, rather than replacing us with posthumans. I agree, and will address this in a subsequent article. For now, note that I’m not equating Team Posthuman with everyone at the AI companies.
He writes:
One topic that often comes up in sci-fi depictions of AI, but that I intentionally haven’t discussed here, is “mind uploading”, the idea of capturing the pattern and dynamics of a human brain and instantiating them in software. This topic could be the subject of an essay all by itself, but suffice it to say that while I think uploading is almost certainly possible in principle, in practice it faces significant technological and societal challenges, even with powerful AI, that likely put it outside the 5-10 year window we are discussing.
What the hell? He wrote this article in 2024, which means he thinks uploaded posthumans might arrive by 2029 — three years from now.
The second part of this is a straightforward extrapolation from MacAskill’s claim about destroying the biosphere being net positive plus the longtermist vision, according to which virtually all future people will be digital in nature, living in vast computer simulations spread throughout the universe. In those simulated environments, we could then recreate forests and meadows, as 1s and 0s, populated by artificial creatures that don’t experience suffering.
Here’s MacAskill in his own perverse words:
Human expansion means that the biomass of wild land mammals has decreased by a factor of seven compared to prehuman times, mostly due to the megafaunal extinctions I discussed in Chapter 2. … On balance, various studies suggest that human activity over the last forty years has probably decreased vertebrate and invertebrate populations, though the evidence is limited and somewhat conflicting. How you evaluate this depends on your view on wild animal wellbeing. It’s very natural and intuitive to think of humans’ impact on wild animal life as a great moral loss. But if we assess the lives of wild animals as being worse than nothing on average, which I think is plausible (though uncertain), then we arrive at the dizzying conclusion that from the perspective of the wild animals themselves, the enormous growth and expansion of Homo sapiens has been a good thing.
“Apocalyptic” is a strong word. But what else would you call a battle for the continued survival of our species?
As I’ve also noted before, becoming an immortal piece of software — if that’s even technologically feasible — would open the door to many horrors beyond human comprehension. For example, it could make eternal dictatorships possible. (Imagine the Singularity happening next year and Donald Trump getting to live forever.) It could enable such dictators to torture political dissidents for trillions of years, until the heat death of the universe. There’s a very long list of agonizing possibilities that mere humans need not worry about.
Again, I am very skeptical that becoming posthuman is even possible. I’m gaming out what would happen if the TESCREAL vision laid out by people like Amodei, Alexander, and Musk were to be successfully implemented.
It needn’t be that people start using the TESCREAL acronym. What matters is the concept this acronym denotes. Once people understand what these transhumanists, singularitarians, Rationalists, EAs, and longtermists are up to, they will be outraged.
Great article, Emile. I do not think these people have a realistic grasp of reality. Humanity is not going to sit idly by while Altman and Musk just do what they want, take their land, take over their countries and destroy humanity. But what continues to baffle me is their belief in "uploading" the human mind as some kind of advancement of the species. I have yet to find anyone in that movement that can explain how anyone would know that a mind upload had actually worked.
Given that chatbots today can pretty realistically mimic human speech and generate realistic facsimiles of actual living or dead humans, how would anyone know if a mind upload was successful? My late brother used to say - "Is it a copy or a move?" IOW, has your actual consciousness really been moved to the cloud or whatever digital substrate they create to house uploaded minds? Or, were "you" just copied, turned into a digital file that seems conscious to everyone else, has all of the characteristics of you but without any of the actual substance of you. You're dead but everyone else...those who haven't been "uploaded", thinks you've been transformed into a digital being. And I submit, there is no way for anyone to know.
Submitting to whatever procedure they devise to upload minds would be like playing Russian Roulette. Maybe you'll live forever. More than likely you just die during the process.
Anyway, appreciate your work.