Why Worry About the ASI Race?

Plus, a (hopefully) interesting history of the field of AI safety. (1,600 words)

Disambiguating a Common Phrase

Since I’m technically on vacation right now (until September 1), I thought I’d publish a relatively short piece on a phrase I hear a lot these days: “the danger of the race to build ASI” (artificial superintelligence). It’s worth noting, I think, that this phrase is ambiguous. It can mean two radically different things.

On the one hand, people in the field of AI safety — many of whom are “doomers”¹ — believe the ASI race is dangerous because it will result in ASI. Specifically, they believe it will likely result in misaligned ASI unless significant steps are taken to ensure that ASI is “value-aligned” (by which they typically mean “aligned with the TESCREAL worldview”). Many of these people will add that ASI is imminent, and hence that everyone on Earth will die in the very near future.

On the other hand, when people like me talk about “the dangers of the ASI race,” we typically mean that trying to build ASI itself poses significant threats to individuals, democratic societies, and perhaps even humanity itself. Many of us don’t believe that ASI is imminent — if it’s even possible to build, assuming the term is meaningful in the first place. But this is beside the point. The fact is that rushing toward what the AI companies call “ASI” has inflicted, and will continue to inflict, serious harms on society and the individuals who comprise it.

Hence, worrying about the dangers of the ASI race doesn’t commit one to a belief that ASI will arrive anytime soon, if ever. By analogy, you could imagine multiple companies racing to build a perpetual motion machine (PMM). They pour huge amounts of money into their respective projects, polluting the environment, exacerbating climate change, destroying communities, etc. in the process. Despite the fact that PMMs are physically impossible, according to the laws of thermodynamics, one could still talk about “the dangers of the race to PMM” — an imaginary end-goal that these companies will never reach.

The same goes for the ASI race. I think this is important to note because referring to “the dangers of the ASI race” could be misunderstood as suggesting that one believes ASI is possible and imminent. One interpretation of the critiques of scholars like Emily Bender, Alex Hanna, Timnit Gebru, etc. is that they are deeply concerned about the ASI race — that is, it’s deleterious consequences — despite them believing (as I understand their views) that ASI isn’t even a coherent concept.

Whenever you hear someone talking about the ASI race being dangerous, it’s worth taking a minute to decipher what they mean. Are they referring to the race itself, or the anticipated outcome of this race?

AI Ethics vs. AI Safety

This ties into another issue worth mentioning, I think. It concerns the distinction between AI ethics and AI safety. One often hears this distinction elucidated as follows: AI ethics focuses on near-term risks whereas AI safety focuses on long-term risks.²

But this is misleading. For example, AI ethicists worry about the environmental consequences of current AI systems — seen by the AI companies as the stepping-stones to ASI. (Hence, to build ASI, we must first build systems like ChatGPT-5, Claude Mythos, etc.) AI has a nontrivial carbon footprint, and indeed the CO2 emissions of companies like Google and Microsoft have risen — despite promises to be carbon-neutral or carbon-negative by 2030 — due almost entirely to AI.

But, of course, climate change is a long-term risk. The CO2 injected into the atmosphere right now will linger for millennia, affecting the livability of Earth for the next 10,000 years — a longer period of time than civilization has so far existed.³

On the flip side, many AI safetyists and doomers believe that ASI could arrive in the next few years, and that, since we don’t yet know how to “align” it with our (i.e., the TESCREAL worldview’s) values, it will annihilate humanity by default. This is clearly a near-term risk, as it would mean the untimely death of you and me, our children and parents, and everyone else alive a few years from now.

A better way of understanding the difference between AI ethics and AI safety is in terms of the distinction made above. Those in the first camp are mostly concerned with the externalities of attempting to build ASI, while those in the second tend to ignore or minimize the inimical effects of the race itself, focusing instead on the “existential” stakes of actually building a superintelligent machine.

Herein lies the most salient difference between these two fields.

History Detour:

AI safety, by the way, is an outgrowth of the TESCREAL movement. It started as a subfield of Existential Risk Studies, which encompasses a wide range of “existential risks” arising from synthetic biology (engineered pandemics), natural pandemics, molecular nanotechnology (gray goo), nuclear war, asteroid and comet impacts, volcanic supereruptions, gamma ray bursts, vacuum decay, and even our simulation getting shut down (lol).

Because many existential risks scholars — such as myself, back in the day — believed that ASI likely poses the most significant near-term existential risk, a subfield gradually coalesced around the subject, with its own unique terminology, hypotheses, norms, and paradigms. A key figure in the founding of AI safety was, of course, Eliezer Yudkowsky, a goofball whose roots in the TESCREAL movement go back to the Extropian movement of the 1990s.

In more detail: Existential Risk Studies emerged as a branch of the TESCREAL movement. The idea was this: to create a posthuman paradise among the stars, we will need to develop advanced technologies. How else will we reengineer humanity and conquer the cosmos? However, such technologies will introduce unprecedented threats — existential risks — to this future. Since we can’t not develop these technologies (because then we’d never reach utopia), the only path forward is to develop synthetic biology, molecular nanotechnology, and ASI as carefully as possible. Existential Risk Studies was founded to study the risks associated with these technologies and devise effective strategies for neutralizing them. This is what my work for some 10 years focused on.

AI safety then arose within Existential Risk Studies because many saw ASI as imminent and particularly dangerous. If we ensure that ASI is “aligned” (with the TESCREAL worldview), then “we” get to live forever in a cosmic utopia marked by “astronomical” amounts of value. But if it’s “misaligned,” then everyone on Earth will perish and, much worse, our posthuman descendants will lose the opportunity to flood our future light cone with “value.”

Of note is that AI safety has recently expanded beyond the TESCREAL movement. Whereas virtually all early practitioners in the field were committed to the TESCREAL worldview, there is a growing number of people working on AI safety projects who aren’t TESCREALists. So far as I can tell, David Kruger provides an example. These people aren’t so much focused on how misaligned ASI would foreclose what Toby Ord calls our “vast and glorious” future as posthumans living in giant computer simulations spread throughout the cosmos, but rather on how misaligned ASI would kill the entire human population. It’s the death of these people — including you and me — that matter to them and thus motivate their AI safety research, not the utopian promise of posthumans in a cosmic paradise.

This is an interesting development. If you agree with me that TESCREALism is a kind of new religion, you might say that the field of AI safety is being secularized.

My Current View

What are my views on the matter? I mentioned above that my primary concern is the harms caused by trying to build ASI. Such harms include:

• exploiting workers in the Global South,

• stealing our intellectual property,

• exacerbating climate change,

• sucking up massive amounts of energy,

• flooding the Internet with slop, disinformation, and deepfakes,

• inducing episodes of psychosis in vulnerable populations,

• perpetuating harmful stereotypes due to algorithmic bias,

• enabling unprecedented levels of mass surveillance,

• working with the US military,

• and so on.

I see no reason for believing that ASI is imminent, and indeed the TESCREAL pro-extinctionist Sam Altman recently acknowledged that the hype around ASI has been overblown. Altman, of course, has been one of the biggest hypesters out there, even arguing last year that the Singularity has already started.

I’m not even sure that the term “ASI” is meaningful, as no one can agree on a definition and the concept of intelligence itself is poorly defined. (To a great extent, our contemporary notion of intelligence has been influenced by the eugenics movement, which has morphed into the TESCREAL movement. Hence, Gebru and I characterize TESCREALism as the most recent iteration of the “eternal return of eugenics.”)

That said, it also seems clear to me — I think — that if the AI companies were to build a system that could outmaneuver us in every important respect, solve complex problems much faster than we can, set and modify its own goals without human intervention, and act as autonomous agents in the world, then we’re in very big trouble. I don’t see any good reason for thinking that our species would survive for very long competing against an artificial system of this sort. You can call that “ASI,” but you don’t need to, to grasp the danger.

Hence, if the ASI race succeeds in building an “ASI,” the outcome would likely be terrible. (Even the “good” outcomes would be terrible, as these would also inevitably lead to the extermination of our species.) But I don’t think the ASI race will “succeed,” at least not anytime soon, and thus when I talk about “the dangers of the ASI race,” I mean “the dangers of trying to build this thing called ‘ASI,’ whatever that is exactly.”

But what do you think? What am I missing? Is this distinction useful? (It is in my own mind!) As always:

Thanks for reading and I’ll see you on the other side!

1. That is, they got into the field of AI safety because they have a high p(doom).

2. For more on the animosity between these fields, see this article.

3. To paraphrase an article that I’m unable to find at the moment.