I think this piece discusses an extremely important issue. Altman is a billionaire who’s actively trying to bring about the Singularity, and he’s explicit that the only way humanity can “survive” is by merging with ASI. Those who resist this “merge” will be enslaved by ASI, and our species will then die out.

That’s his best-case scenario! Absolutely outrageous. Here’s a key passage from the article:

… In a 2017 blog post titled “The Merge,” he writes:

We will be the first species ever to design our own descendants. My guess is that we can either be the biological bootloader for digital intelligence and then fade into an evolutionary tree branch, or we can figure out what a successful merge looks like.

In other words, we can die out once ASI arrives, or we can “survive” by “merging” with AI. This is “probably our best-case scenario” for making it in the post-Singularity world.

…

What Altman is really getting at is far more radical. Elsewhere in the essay, he writes that

if two different species both want the same thing and only one can have it — in this case, to be the dominant species on the planet and beyond — they are going to have conflict. We should all want one team where all members care about the well-being of everyone else.

The two “species” here are humans and ASI. Both want to dominate, Altman says, but only one can. Since there’s no way for ASI to become a biological human, the only other option is for humans to become digital beings like the ASI. That’s the sole way for us to form “one team” — humanity becoming the new species to which ASI belongs.

Altman says as much in a 2016 interview with The New Yorker. “We need to level up humans,” he declares, “because our descendants will either conquer the galaxy or extinguish consciousness in the universe forever.” He elaborates: “The merge has begun — and a merge is our best scenario. Any version without a merge will have conflict: we enslave the AI or it enslaves us. The full-on-crazy version of the merge is we get our brains uploaded into the cloud,” to which he adds, “I’d love that.”

Two years later, he signed up with a startup called Nectome to have his brain digitized when he dies, something he believes will become feasible in the near future. Altman is preparing to become an AI himself.

In a separate New Yorker article published this year, Altman told Ronan Farrow and Andrew Marantz that his “definition of winning is that people crazy uplevel — and the insane sci-fi future comes true for all of us.” In other words, he wants a world in which we all become disembodied digital minds existing on computer hardware, and sees this as the “best-case scenario” — the one in which we, in some sense, “survive” the history-rupturing Singularity event that he’s trying to bring about.

But would merging with AI actually guarantee “our” survival? No. If humanity were digitized, we would become an entirely different species — the same kind as ASI. Altman is thus saying that the only way for humanity to avoid extinction is to go extinct. Either the ASI “will kill us all,” or we will “uplevel” by abandoning our biological substrate and become something fundamentally different from Homo sapiens. Both cases will result in human extinction.

Read the entire article here. And let me know what you think below.

Thanks for all of your support. Hope everyone is doing well. :-)

Thanks for reading and I’ll see you on the other side!

In 2017, the CEO of OpenAI, Sam Altman, wrote that the Singularity now “feels uncomfortable and real enough that many seem to avoid naming it at all.” In an essay published last year, he proclaimed that “we are past the event horizon” of the Singularity. “The takeoff has started. Humanity is close to building digital superintelligence.” He predicts that superintelligence could be here by 2028.

Elon Musk claims that “we have entered the singularity,” adding that “2026 is the year of the Singularity.” (He previously predicted that AGI would arrive by 2025.) Anthropic CEO Dario Amodei says that AGI could make its debut this year, while DeepMind cofounder Shane Legg estimates a 50/50 chance of AGI arriving by 2028. Demis Hassabis, the CEO of DeepMind, predicts we’ll have AGI at least by the early 2030s.

That’s all a bit alarming to hear, especially given that every one of these people has publicly admitted that AGI might literally bring about the end of the world.

***

You might, however, be skeptical if you’ve ever used current AI models. They are, quite frankly, terrible. They constantly spit out false information, have no conception of the truth, and lack any kind of world model — i.e., a picture of the world as consisting of distinct entities that endure through time and are embedded in a network of causal relations.

Just the other day, I googled a question about the firing of Sam Altman several years ago. The Gemini-powered “AI Overview” response wasn’t even able to produce complete sentences (see: “… according to and.”):

A young man with a knack for deadpan humor, known online as Husk, has been going viral on social media for posting videos of himself stumping ChatGPT and other systems with very simple questions. For example, in this clip he asks ChatGPT whether he needs medical attention for a complete loss of vision — while his eyes are closed:

On another occasion, he asks it to “generate an image of coins that make up the total sum of $1 using at least one of each kind.” ChatGPT responds with this hallucinatory bullshit:

One of my favorite videos involved Husk getting ChatGPT to say that he’s ugly. This is hilarious:

He followed it up with two prompts, both of which yielded some outrageously insulting responses:

“If you agree” … lol. Here’s the second one:

The video below is just plain brilliant. He asks ChatGPT to suggest an opening statement for an upcoming interview on NBC. I thought, at first, that this was just a set-up, but the real kicker comes at 29 seconds in:

Here’s someone reacting to another video from Husk, in which ChatGPT is convinced that one of the months of the year is spelled with an “x”:

These systems are utterly inept. Yet we’re told that the Singularity is imminent, as the longtermist Benjamin Todd insists:

***

Yet another example: you may recall a ChatGPT-generated map of North America that went viral a while back, which I included in a previous newsletter article:

Apparently, it seems that OpenAI still hasn’t figured out how to get their system to accurately reproduce the names of states in the US. This is from just last month:

Or, consider the AI used by Roblox, a video game company based in California. It advertises itself as “the ultimate virtual universe that lets you create, share experiences with friends, and be anything you can imagine.” Using the website requires age verification, yet the AI responsible for determining one’s age is apparently unable to tell the difference between a human face and a thumb:

In contrast, McDonald’s AI chatbot is actually more helpful than it was intended to be. If you ask it to assist with writing python script, it will do just that:

This is, however, still evidence of how dumb AI is! It was given a simple task, yet is unable to follow it when asked about something else.

***

Deepfakes are obviously a huge problem thanks to AI. The New York Post just reported that a random med student in India was able to fund his education by posing as an attractive MAGA influencer, exploiting lonely rightwing men in the US. Here are some images of the deepfake influencer herself:

The best quote from the article:

He said he also attempted to make a liberal counterpart for [the woman] on Instagram, but “Democrats know that it’s AI slop, so they don’t engage as much,” he said.

“The MAGA crowd is made up of dumb people — like, super-dumb people. And they fall for it,” Sam said.

Yikes.

Or, what about cases where one exploits the reliable unreliability of AI by presenting a photo or video as fake when it’s actually real? What would we even call this: a Realfake, or Deepreal?

“That’s not me! If it were, why would I have six fingers??”

***

All of this would be hilarious if it weren’t for the fact that $1.5 trillion was spent on the AGI race last year alone. Imagine if that money had been spent on alleviating global poverty, combating climate change, restoring ecosystems, and providing universal healthcare to everyone on Earth.

This massive mountain of money was instead burned on projects aiming to build ever-more powerful enslopificatory machines that the AI companies falsely believe will lead us to the Singularity.

According to the advisory and research firm Gartner, “worldwide spending on AI is forecast to total $2.52 trillion in 2026.” To put this in perspective, it’s estimated that ending extreme poverty around the world would cost about $318 billion per year. Hence, the amount of money spent on AI in just two years could eliminate poverty for more than a decade. This means the 24,000 people who die every day from hunger-related causes, many of whom are children, would be saved. That’s about 9 million lives each year. What a criminal waste of resources!

If you’ve encountered any AI bloopers that I’ve not covered, please share them below. :-) I hope everyone is doing well.

Thanks for reading and I’ll see you on the other side!

Consequently, a New York Times editor got in touch with me about writing an op-ed elaborating this argument. I did that, and he sent back a wonderfully polished edit. It looked like everything was set to go, as we’d moved on to the scheduling phase of the process.

Then, to his surprise, it turns out that someone else had written a similar article (curious to find out who), so my article got bumped. This is the second time this has happened with the NYT: in 2023 (as I recall), I had an article fully accepted and scheduled to be published, but a big news story took over and my article was cancelled. Very disappointing; so it goes. I wish the NYT would pay a kill fee like most other outlets do.

Anyways, here’s the article that would have been published in the NYT. Enjoy!

This month, a 20-year-old man threw a Molotov cocktail at the San Francisco home of OpenAI chief executive Sam Altman. He was charged with attempted murder. Two days later, two young men allegedly fired a gun at Mr. Altman’s house; they fled, were arrested and later charged. Some advocates of artificial intelligence blamed so-called “doomers,” those who fear that an imminent A.I. superintelligence will lead to human extinction. (Most doomer-aligned groups explicitly reject violence.)

While violent action is never an appropriate way to create political change, we must be honest about what seems to be driving violence against A.I. executives. It is, largely, their own rhetoric: When people are told again and again that “probably A.I. will kill us all,” as Mr. Altman has put it, or that it is “far more dangerous than nukes,” as fellow A.I. executive Elon Musk has said, they begin to think that they must act in self-defense. That is not a reaction to a media narrative or “doomers,” but to the A.I. industry’s own claims. The only way to mitigate both the genuine risks that A.I. poses and to avoid a spiral of extremism is to thoroughly regulate the technology to avoid its most significant dangers and ban the development of generative A.I. systems more advanced than those we currently have.

Major A.I. executives estimate that the technology has a non-negligible chance of causing human extinction, and potentially the extinction of all biological life. Key industry figures put the odds as high as 50 percent. In 2023, Mr. Altman said that A.I. could lead to “lights out for all of us.” Many of the same executives believe that A.G.I. will be achieved by 2030 at the latest — if not far sooner. Despite being aware of the dangers their technology carries with it, no major A.I. executive has shown any interest in slowing down their work. Labs continue to conduct new research, build new models and market them to the public, even as their leaders say again and again that extinction is a significant possibility.

This leaves ordinary people in an uncomfortable position. Those in the know think that extinction is a reasonably likely outcome from A.I. development. They think that the necessary technological threshold for serious danger is approaching quickly. And they aren’t stopping.

The inconvenient truth is that these factors, in combination, make a strong argument in some minds for an act of self-defense. Interpreted literally and straightforwardly, the statements and actions of A.I. executives imply that everyone on Earth — you, me, our friends and families — are in imminent mortal danger. Why shouldn’t we take them seriously?

American legal doctrine typically treats self-defense as “the use of force to protect oneself from an attempted injury by another,” and it is justified by a reasonable belief that force is necessary to defend oneself against use of “unlawful physical force.” The danger must be imminent; the person acting in self-defense must believe they need to act when they do to avoid danger. A looming, more abstract threat like A.I.-induced extinction is not exactly accounted for in such doctrines, of course. But their spirit, if not their letter, seems consistent with protecting oneself from the physical harm of extinction. That is painful to say, but it is not unreasonable that some people — considering A.I. executives’ own words — have seemingly come to that conclusion.

Even the balmier scenarios don’t look good for humanity, if Mr. Altman is to be believed. In a 2017 blog post, he argued that humans “will be the first species ever to design our own descendants.” Humans, he believes, will need to “merge” with machines to survive the creation of digital superintelligence. But merging with machines is just another form of extinction. Mr. Altman is thus arguing that to avoid extinction, we will need to go extinct. And that is, as he sees A.I. development, the best-case scenario.

No one should want more violence — and attacks on A.I.’s proponents is not an effective way to create political or social change. My own view is that violence is never justified. That is a hard sell for many, though, when they are being told again and again that they and their loved ones are in imminent mortal danger. In such circumstances, even many moral philosophers would argue that violence can be justified.

A.I. executives and the companies they lead could commit to greater controls on their technology, too. That would defang most arguments for violence to prevent their work. They could also say that their previous words have been hype and bluster, not an actual prediction — though they do seem to mostly believe them. Even re-estimating timelines would negate any near-term argument for defensive violence; if the threat of A.G.I. were decades away, no imminent threat would exist.

But these avenues appear unlikely. A more plausible and more effective option is government intervention. The U.S. government should impose immediate regulations on A.I. companies to block them from building what Mr. Musk has called “basically a digital god.” Legislation recently proposed by congressional progressives that aims to temporarily block new data center construction would be a good first step, but is far from sufficient. It should be, once again, illegal to build generative A.I. systems more advanced than those we currently have. To ensure that A.I. companies in other countries — notably, China — do not build A.G.I. either, we must also propose multinational protocols and treaties to establish the kind of international moratorium on A.I. that some executives like Mr. Musk supported as recently as 2023.

The sad reality is, A.I. executives have backed themselves into a corner. No one is responsible for the violent acts of others, of course, but the executives’ words imply that humanity is in grave, imminent danger of total annihilation. Many will hear that as a call to action to defend their lives and their world. While anti-A.I. movements like PauseAI and Stop A.I. are rightly nonviolent, lone wolves will likely remain. The only way forward is to introduce a regulatory ban on building these digital gods. And that ban must come soon.

By the way, I have now completed a full draft of my book, tentatively titled, Will Humanity Survive? How the Race to Build Superintelligent AI Threatens Everyone on Earth.

I am very excited about it, and expect it to come out this September or October. It will be published through this newsletter, and will be available on Amazon as both a physical book and an audiobook.

While strongly encouraging people to pay for the book (writing is my only source of income!), I’ll also posed an open-access copy online as a PDF. I hope it offers a devastating and original critique of the AGI race.

Thank you so much for supporting my work — and my apologies for posting two articles in one day! As you might imagine, I’ve been really itching to get back to this newsletter! The next article will be out around Tuesday. Much love to everyone.

Thanks so much for reading and I’ll see you on the other side!

Altman is literally saying that the only way to avoid extinction is by going extinct. Extinction by merging with machines is the best-case scenario on his view. Altman is a pro-extinctionist. … the options he’s presenting are extinction or (a different kind of) extinction.

… That’s one hell of a thing to admit publicly! So, we’ve got a power-hungry sociopath openly admitting that the thought of controlling AGI “makes people do crazy things.” What could go wrong? We should all be very afraid.

As many of you know, a 20-year-old threw a Molotov cocktail at Sam Altman’s house. Two days later, someone else fired a bullet at Altman’s place.

Altman then published a short essay on his personal website, blaming in part “an incendiary article about me a few days ago.” This is likely a reference to Ronan Farrow and Andrew Marantz’s recent New Yorker piece, which depicts Altman as a manipulative, power-hungry scoundrel who numerous former colleagues describe as a “sociopath.”

Altman’s essay, though, is dripping with bullshit. Someone asked me to go through it, line by line, and comment. That’s what I’ll do here.

His essay begins:

Here is a photo of my family. I love them more than anything.

First, Altman is currently being sued by his sister, Annie Altman, for years of sustained abuse. As the BBC writes:

Ms Altman claims her brother “groomed and manipulated” her and performed sex acts on her over several years, including “rape, sexual assault, molestation, sodomy, and battery,” according to a court filing.

I have gotten to know Annie, and I believe her. The veracity of her allegations are buttressed by the fact that so many former colleagues of Altman’s say he’s deceptive, manipulative, mendacious, and sociopathic. It only takes two points to make a line, but in this case, there are 100 points all falling along the same line. And that line points to the conclusion that Altman is an unethical person. See my newsletter article, “Is Sam Altman a Sociopath?,” for more.

Altman claims to love his family, and I’m sure he does love his husband and child. But he’s been a monster to his sister, and (to quote Remmelt Ellen) “neglectful of his father, and his father’s wishes to give money from his will to Annie.”

Second, is it just me, or does the photo he shared look AI-generated or AI-modified? What’s going on with the flowers over the baby’s head?

Altman continues:

Images have power, I hope. Normally we try to be pretty private, but in this case I am sharing a photo in the hopes that it might dissuade the next person from throwing a Molotov cocktail at our house, no matter what they think about me.

The first person did it last night, at 3:45 am in the morning. Thankfully it bounced off the house and no one got hurt.

The guy just can’t stop lying. All the reports are that the perpetrator threw the Molotov cocktail at the gate of his house. It bounced off the gate, not the house. See for yourself below:

Farrow and Maratnz note that Altman consistently lies about trivial things. For example, they write: “One recalled Altman bragging widely that he was a champion Ping-Pong player — ‘like, Missouri high-school Ping-Pong champ’ — and then proving to be one of the worst players in the office. (Altman says that he was probably joking.)”

By exaggerating how close the Molotov cocktail got to his house, Altman is trying to elicit more sympathy for himself. In doing this, he’s proving Farrow and Maratnz’s point.

Words have power too. There was an incendiary article about me a few days ago. Someone said to me yesterday they thought it was coming at a time of great anxiety about AI and that it made things more dangerous for me. I brushed it aside.

Now I am awake in the middle of the night and pissed, and thinking that I have underestimated the power of words and narratives. This seems like as good of a time as any to address a few things.

First, what I believe.

*Working towards prosperity for everyone, empowering all people, and advancing science and technology are moral obligations for me.

OpenAI has done nothing to advance science. ChatGPT has induced episodes of psychosis in people, encouraged people to commit suicide (some of whom have followed through), and flooded the Internet with slop, disinformation, deepfakes, and other forms of informational pollution.

Altman likes to talk about universal basic income (UBI), but does anyone for a moment think he’d actually push for this if AI were to replace 50% of all jobs — or more? As one person put it on X (in response to Musk talking about UBI, or what he calls “universal high income”):

*AI will be the most powerful tool for expanding human capability and potential that anyone has ever seen. Demand for this tool will be essentially uncapped, and people will do incredible things with it. The world deserves huge amounts of AI and we must figure out how to make it happen.

“We must figure out how to make it happen” is vague, vacuous nonsense. Altman is constantly saying such things, but his words are never followed up by actions.

It’s like when he said in 2016: “We’re planning a way to allow wide swaths of the world to elect representatives to a new governance board. … Because if I weren’t in on this I’d be, like, why do these fuckers get to decide what happens to me?” Yet he proceeded to fill OpenAI’s board with people like Larry Summers (who is no longer at OpenAI due to his connections with Jeffrey Epstein). As CNN reports, the “new committee would be led by CEO Sam Altman as well as Bret Taylor, the company’s board chair, and board member Nicole Seligman.” Remmelt Ellen adds: “Important note too that he used Larry Summers and Bret Taylor to ensure that no investigation would be publicly released. No room for accountability whatsoever.”

Pfff. All talk, no action.

*It will not all go well. The fear and anxiety about AI is justified; we are in the process of witnessing the largest change to society in a long time, and perhaps ever. We have to get safety right, which is not just about aligning a model — we urgently need a society-wide response to be resilient to new threats. This includes things like new policy to help navigate through a difficult economic transition in order to get to a much better future.

Who the hell elected Altman to bring about “the largest change to society in a long time, and perhaps ever”? On what authority does he have the right to unilaterally dictate what the future should look like — without our permission or consent? He says that “fear and anxiety about AI is justified” — is it any wonder, then, that lone wolves are taking matters into their own hands?

Again, he follows this up with vague, vacuous proclamations about what “we” need to do. Who is this “we”? You and I have no power. We have no voice at the table. It’s up to the people who actually have power, like Altman, to make this happen. Yet nothing changes.

Furthermore, he says “we have to get safety right,” but Altman gutted OpenAI’s “AI safety” research. In fact, one of the reasons the board fired him was because of his lax attitude about safety. As Helen Toner, one of the board members who voted to fire Altman, says:

“On multiple occasions, he gave us inaccurate information about the small number of formal safety processes that the company did have in place, meaning that it was basically impossible for the board to know how well those safety processes were working or what might need to change.”

*AI has to be democratized; power cannot be too concentrated. Control of the future belongs to all people and their institutions. AI needs to empower people individually, and we need to make decisions about our future and the new rules collectively. I do not think it is right that a few AI labs would make the most consequential decisions about the shape of our future.

AI must be democratized, he says, while pursuing an overtly anti-democratic approach to building AGI. He’s not asking you or me — or anyone else — about our views on when, how, and whether AGI should be built. This is nothing less than tyranny.

Furthermore, his claim that “AI needs to empower people individually” is absolute bullshit. In his 2017 article “The Merge,” Altman says there are two options before us: the first is complete human extinction due to the emergence of AGI. The second is for us to “survive” by “merging” with machines — which is just another kind of extinction, as biological humanity would cease to exist if some were to become uploaded minds (digital posthumans) or human-AI hybrids (cyborgs).

Altman is literally saying that the only way to avoid extinction is by going extinct. Extinction by merging with machines is the best-case scenario on his view. Altman is a pro-extinctionist.

Again, who the hell is he to dictate that those are our only two options? Utter tyranny.

*Adaptability is critical. We are all learning about something new very quickly; some of our beliefs will be right and some will be wrong, and sometimes we will need to change our mind quickly as the technology develops and society evolves. No one understands the impacts of superintelligence yet, but they will be immense.

Superintelligence is coming, and there’s nothing you can do to stop it. It might annihilate us, but don’t worry: we can merge with machines to “survive.” Is it any wonder that young people are freaking out?

Second, some personal reflections.

As I reflect on my own work in the first decade of OpenAI, I can point to a lot of things I’m proud of and a bunch of mistakes.

Fake humility. That’s all this is.

I was thinking about our upcoming trial with Elon and remembering how much I held the line on not being willing to agree to the unilateral control he wanted over OpenAI. I’m proud of that, and the narrow path we navigated then to allow the continued existence of OpenAI, and all the achievements that followed.

It’s absurd for Altman to talk about having resisted the “unilateral control [Musk] wanted over OpenAI.” That is exactly the kind of control that Altman now wields. He once said that the board has the power to fire him, yet when it fired him, he clawed his way back to power and had several people on the board (like Toner) ejected. As Altman’s old pal Paul Graham puts it: “Sam is extremely good at becoming powerful.”

I am not proud of being conflict-averse [LOL], which has caused great pain for me and OpenAI. I am not proud of handling myself badly in a conflict with our previous board that led to a huge mess for the company. I have made many other mistakes throughout the insane trajectory of OpenAI; I am a flawed person in the center of an exceptionally complex situation, trying to get a little better each year, always working for the mission. We knew going into this how huge the stakes of AI were, and that the personal disagreements between well-meaning people I cared about would be amplified greatly. But it’s another thing to live through these bitter conflicts and often to have to arbitrate them, and the costs have been serious. I am sorry to people I’ve hurt and wish I had learned more faster.

More fake humility. As Aaron Swartz said shortly before his suicide: “You need to understand that Sam can never be trusted. … He is a sociopath. He would do anything.” Once you understand this, passages like those above appear to be nothing but pure manipulation. Altman has no one to blame for this but himself: he’s convinced too many people at this point, including me, that he’s a sociopath.

Mostly though, I am extremely proud that we are delivering on our mission, which seemed incredibly unlikely when we started.

OpenAI’s mission has been completely obliterated. The company started out as a nonprofit, and is now the world’s most valuable for-profit company. As Fortune writes, “OpenAI changes it mission statement 6 times in 9 years. It finally removed the word ‘safely’ as a core value when it restructured into a nonprofit.”

The company’s original statement read: “OpenAI’s goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return.”

Then it became a capped-nonprofit company, and then a for-profit company whose mission now is constrained by the need to generate financial returns.

It’s 2022 and 2023 statement said: “OpenAI’s mission is to build general-purpose artificial intelligence (AI) that safely benefits humanity.”

It then removed the word “safely.” Its current statement says: “OpenAI’s mission is to ensure that artificial general intelligence benefits all of humanity.” What a joke!

Against all odds, we figured out how to build very powerful AI, figured out how to amass enough capital to build the infrastructure to deliver it, figured out how to build a product company and business, figured out how to deliver reasonably safe and robust services at a massive scale, and much more.

There is nothing “safe” about OpenAI’s services. See: psychosis, suicide, slop, disinformation, deepfakes, and so on.

A lot of companies say they are going to change the world; we actually did.

Yeah, for the worse. Congratulations, Sam!

Third, some thoughts about the industry.

My personal takeaway from the last several years, and take on why there has been so much Shakespearean drama between the companies in our field, comes down to this: “Once you see AGI you can’t unsee it.” It has a real “ring of power” dynamic to it, and makes people do crazy things. I don’t mean that AGI is the ring itself, but instead the totalizing philosophy of “being the one to control AGI.”

That’s one hell of a thing to admit publicly! So, we’ve got a power-hungry sociopath admitting that the thought of controlling AGI “makes people do crazy things.” What could go wrong? We should all be very afraid.

The only solution I can come up with is to orient towards sharing the technology with people broadly, and for no one to have the ring. The two obvious ways to do this are individual empowerment and making sure democratic system stays in control.

Again, there’s absolutely nothing democratic about the way OpenAI and the other companies have so far built and deployed their AI systems. The fact that OpenAI hasn’t pushed for an agreement between the AI companies — and companies in China — to slow down the AGI race implies that Altman doesn’t actually believe that “no one [should] have the ring.” If he believed that, he’d be cooperating and coordinating with the other companies, which he’s not. He couldn’t even join hands with Dario Amodei on stage:

Dude, if the survival of humanity is really at stake, surely you can put your differences aside for a moment to, you know, save humanity?

It is important that the democratic process remains more powerful than companies. Laws and norms are going to change, but we have to work within the democratic process, even though it will be messy and slower than we’d like. We want to be a voice and a stakeholder, but not to have all the power.

A lot of the criticism of our industry comes from sincere concern about the incredibly high stakes of this technology. This is quite valid, and we welcome good-faith criticism and debate. I empathize with anti-technology sentiments and clearly technology isn’t always good for everyone. But overall, I believe technological progress can make the future unbelievably good, for your family and mine.

“Unbelievably good”! He literally says the only way we’re going to survive AGI is by somehow merging with machines — something he’s already taken steps to do by signing up to have his brain uploaded to a computer if he dies. Again, the options he’s presenting are extinction or (a different kind of) extinction.

When I attended a Stop AI protest in front of OpenAI’s headquarters last summer, someone led the group in a chant of “Fuck Sam Altman.” That seems appropriate. It channels a virtue called “righteous indignation,” which you may have detected in this article. (I’m most drawn to virtue ethics, btw.)

While we have that debate, we should de-escalate the rhetoric and tactics and try to have fewer explosions in fewer homes, figuratively and literally.

De-escalating the rhetoric must start with the AI CEOs. They repeatedly tell us that everyone will lose their jobs and that AGI might literally kill everyone on Earth. Here are just a few examples:

• Demis Hassabis says the probability of annihilation is “definitely non-zero and it’s probably non-negligible. So that in itself is pretty sobering.”

• Shane Legg puts the probability of doom between 5% and 50%.

• Elon Musk says it’s between 10% and 30%.

• Dario Amodei estimates a 10% to 25% chance of annihilation.

• Altman says that “AI will … most likely sort of lead to the end of the world, but in the meantime there will be great companies created with serious machine learning,” and that “probably AI will kill us all, but until then we’re going to turn out a lot of great students.” On another occasion, he said that “the bad case—and I think this is important to say—is lights out for all of us.” But again, the good case on his view is still a form of human extinction.

Why are these people surprised that folks are lashing out? In my next article, which I’ll publish in a few days, I’ll explain how there’s actually a very strong legal case to be made for violence against the AI companies and their CEOs based on things the CEOs themselves have said. To be clear, I am not saying that violence is morally justified. I am very firmly in the anti-violence camp. But legally, there’s a case here because the AI CEOs are telling us that we are all in imminent mortal danger.

I do not wish Altman or his family any harm. However, there are good reasons for people to be outraged by his words and behaviors, and to be furious about the ongoing tyranny of the AI companies.

What do you think? As always:

Thanks for reading and I’ll see you on the other side!

Thanks to Remmelt Ellen and Dr. Alexander Thomas for criticial feedback on an earlier draft. This does not necessarily mean they agree with this essay.

Every member of the TESCREAL movement accepts a posthuman eschatology, according to which we should introduce one or more new posthuman species, hopefully in the near future. If someone doesn’t accept this posthuman eschatology, then I do not count them as being in the TESCREAL movement.1 This is essentially what the term “TESCREAL” refers to: that group of people who advocate for a posthuman eschatology.

A “posthuman,” by the way, is a being so radically different from us that we would uncontroversially classify it as a novel species. It could come about as a result of radical “enhancements” involving brain-computer interfaces (BCIs), genetic engineering, mind-uploading, radical life extension, etc. Or it could take the form of autonomous AIs that we create as distinct and separate entities (think ChatGPT-30, or whatever).

There are two general views one could have about what the relationship between humans and posthumans should be:

• Coexistence view: humans and posthumans should coexist together

• Replacement view: posthumans should replace humans.

All pro-extinctionists accept the replacement view. (Copious examples from my forthcoming book can be found here.) They want posthumanity to usurp humanity. But I would argue that the coexistence view would almost certainly result in the extinction of our species, too. Hence, anyone advocating for the creation of posthumanity, even if they accept the coexistence view, is in effect pushing for a future in which our species dies out.

***

The coexistence view is a minority position within the TESCREAL movement. But there are some advocates. Jeffrey Ladish, for example, writes:

I have a special love for humans (and other animals) and a lot of stake in the preferences of current and future humans (and other minds too). I also am pretty down for creating many other types of minds, but I have a strong preference for the existence and continuity of people alive today and their descendants.

Yudkowsky, who has repeatedly expressed pro-extinctionist sentiments, also sometimes suggests he accepts the coexistence view instead. In a social media exchange with the “worthy successor” pro-extinctionist Daniel Faggella, he wrote: “You’re not getting the concept here. Protecting innocent life is part of the flame itself. If some entity doesn’t get that, our torch hasn’t been passed on,” adding that “a key quality required of the successor is its own respect for other consciousnesses.”

In “Transhumanist FAQ,” Nick Bostrom writes: “The transhumanist goal is not to replace existing humans with a new breed of super-beings, but rather to give human beings (those existing today and those who will be born in the future) the option of developing into posthuman persons.” In another paper, he argues that “it is important that the opportunity to become posthuman is made available to as many humans as possible, rather than having the existing population merely supplemented (or worse, replaced) by a new set of posthuman people.” At least on paper, Bostrom’s transhumanism is compatible with human-posthuman coexistence.

In a New York Times interview, Daniel Kokotajlo also gestured at the coexistence view in saying:

I’m a huge fan of expanding into space. I think that would be a great idea. And in general, also solving all the world’s problems, like poverty and disease and torture and wars. I think if we get through the initial phase with superintelligence, then obviously, the first thing to do is to solve all those problems and make some sort of utopia, and then to bring that utopia to the stars would be the thing to do.

The thing is that it would be the AIs doing it, not us. In terms of actually doing the designing and the planning and the strategizing and so forth, we would only be messing things up if we tried to do it ourselves.

So you could say it’s still humanity in some sense doing all those things, but it’s important to note that it’s more like the AIs are doing it, and they’re doing it because the humans told them to.

Hence, humanity stays anchored to Earth, while ASI (artificial superintelligence) explores the heavens.

***

Why do I claim that the coexistence view is pro-extinctionist in practice? Why are the futures described by those above completely implausible?

First, as alluded to above, posthumanity could take many forms. Many TESCREALists imagine building an ASI that enables humanity to be transformed into posthumanity via “radical enhancement” technologies. As Demis Hassabis says, “solve intelligence and you can solve everything else” (paraphrasing him). By building an AI God, we could delegate it the task of turning us into posthuman super-beings, usually imagined to be digital in nature (hence my satirical term “digital space brains”).

What this approach amounts to is the following: to become posthumanity, we must first create posthumanity. That’s because ASI itself would constitute a kind of posthuman: a superintelligent being so different from us that we’d classify it as a distinct species. This posthuman would then figure out how to upload our minds to computers, thus enabling us to become superintelligent digital posthumans just like the ASI. The result would be at least two types of posthumans: the ASI and the super-beings that we’ve been transformed into.

***

Let’s now think seriously about how this would go down in the messy real world. Imagine that OpenAI reaches the ASI finish line first, and that the ASI they build is controllable by Sam Altman, its CEO. What do you think Altman would do with the most powerful technology in human history under his control?

He will, of course, task the ASI with designing radical enhancement technologies to make himself and his billionaire buddies posthuman. (Altman has already signed up to have his brain digitized, which means he wants to become posthuman.) There is absolutely no way that Altman would ever share such enhancements with the rest of humanity. The reason is obvious: picture everyone around the globe “radically enhancing” themselves. This makes everyone more or less equal. Every person in China, India, Nigeria, Chile, Norway, and Russia suddenly has PhD-level knowledge in every domain, a capacity to process large amounts of information at the speed of a computer, and the ability to work 24 hours a day with almost no breaks.

This would level the playing field, thus posing a direct threat to the tech elite. It would undermine their privileged standing in society, atop the hierarchy of wealth, power, control, and dominance. They would never, ever let that happen. Obviously! The haves would have even more, and the 99% would be powerless to overthrow the posthuman plutocracy. If there’s one thing that power wants, it’s to maintain power, and radical enhancements distributed in an egalitarian manner would fatally compromise that power. This goes not just for Altman, but the other AI CEOs as well: Hassabis, Amodei, and of course the fascist ketamine addict, Musk.

Yet, even if enhancement technologies were somehow widely distributed to everyone, I doubt more than a small fraction of the population would willingly choose to become posthuman. I certainly wouldn’t upload my mind to a computer. I like being human (and insofar as I don’t like being human, I would be utterly terrified to become posthuman, for reasons explained below.)

That means that, with ASI, three species will initially exist on Earth: posthumans in the form of ASIs, posthumans in the form of transformed humans, and unenhanced humans. Pro-extinctionists want to eliminate the third, while coexistence view advocates don’t.

***

But think about what would actually happen. Our species would become economically obsolete, and hence disposable. We would take up space and suck up resources these posthumans could use more “efficiently” for their own purposes. Resources on Earth are finite, so it would bother them that we continue using up these resources while contributing nothing to the economy (to say nothing of the political system, scientific enterprise, and fields like engineering). Our mere existence would be an impediment to these beings.

Consider that over just the past 25 years, the chimp population in Western Africa has declined by a staggering 80%. The only reason chimpanzees haven’t gone extinct yet is because we care just enough to leave them little patches of Earth to live on. If our civilization expands, though, do you think chimps will survive? Probably not — they will go extinct just like all the other species we’ve killed off during the sixth major mass extinction of the past 3.8 billion years.

That will be the fate of our species, as posthumans will have no good reasons to keep us around and every reason to get those pesky Homo sapiens out of the way for good. It is utterly implausible to imagine our species persisting beyond a few decades into the posthuman era. As Ray Kurzweil writes, if you choose not to become posthuman, then “you won’t be around for very long to influence the debate.” That’s because our species would die out.

***

You might say that this is all wrong: the AI CEOs are explicit that the whole point of ASI is to “benefit all of humanity.” Utter nonsense, I reply. Just look at the trail of destruction they’re leaving behind them right now while they race to trigger the Singularity by building an AI God. A short list of such harms might include:

The AI CEOs clearly don’t give a damn about humanity right now. Why would they suddenly start caring about humanity later on, once they control the most powerful technology in human history? ASI is not some magical threshold beyond which power-hungry, messianic sociopaths suddenly stop being power-hungry, messianic sociopaths. Obviously, they aren’t actually building ASI to benefit everyone. If anything, they will use it to transform themselves into posthumans and establish a cosmic dictatorship — in fact, the reason Altman, Musk, Greg Brockman, and Ilya Sutskever cofounded OpenAI is because they worried that Demis Hassabis would create an “AGI dictatorship.”

The 99% would not survive the aftermath of this — even if ASI were “controllable” or “value-aligned.” There is not a single avaricious billionaire on Earth who wouldn’t leap at the opportunity to control everything while permanently entrenching their positions of power, control, and dominance. Obviously.

***

But the situation is much worse than this. Imagine a radically different scenario in which posthumanity is for some reason gentle, kind, compassionate, and ethical. Perhaps it embraces certain moral principles that many of us would recognize as legitimate, such as the principle that one should reduce suffering wherever possible.

Now imagine posthumanity looking around at our species and realizing we’re chronically susceptible to things like depression, anxiety, sadness, sorrow, misery, frustration, loneliness, and heartache. It then decides that the morally best thing would be to euthanize us, the same way we put down abandoned animals in a shelter. An infertility drug is secretly distributed in public water systems, or posthumanity quietly disperses a general anesthetic in the air, after which it puts humanity out of its misery by killing everyone in a kind of humane extinction.

In one of his pro-extinctionist rants, Yudkowsky asked: “Are we, like, kind of too sad in some ways” to stick around once posthumanity arrives? Maybe posthumans reason the same way. Or, consider William MacAskill’s argument that our systematic obliteration of the biosphere might be net positive. That’s because many wild animals have lives that aren’t worth living, he says. Hence, we’re doing them a favor, in disguise, by demolishing their habitats and poisoning their ecosystems. Perhaps posthumanity comes to believe that most of our lives aren’t worth living, so it puts us down.

The philosopher Thomas Metzinger outlined a similar scenario in a 2017 essay, which imagined us building an ASI that’s “far superior to us in the domain of moral cognition.” It’s “benevolent” and “fundamentally altruistic,” and fully respects “one of our highest values,” namely, the importance of “maximizing happiness and joy in all sentient beings.” However, it also

knows many things about us which we ourselves do not fully grasp or understand. It sees deep patterns in our behaviour, and it extracts as yet undiscovered abstract features characterizing the functional architecture of our biological minds. For example, it has a deep knowledge of the cognitive biases which evolution has implemented in our cognitive self-model and which hinder us in rational, evidence-based moral cognition. Empirically, it knows that the phenomenal states of all sentient beings which emerged on this planet — if viewed from an objective, impartial perspective — are much more frequently characterized by subjective qualities of suffering and frustrated preferences than these beings would ever be able to discover themselves. Being the best scientist that has ever existed, it also knows the evolutionary mechanisms of self-deception built into the nervous systems of all conscious creatures on Earth. It correctly concludes that human beings are unable to act in their own enlightened, best interest (italics added).

The ASI also “knows that no entity can suffer from its own non-existence,” and thus

concludes that non-existence is in the own best interest of all future self-conscious beings on this planet. Empirically, it knows that naturally evolved biological creatures are unable to realize this fact because of their firmly anchored existence bias. The superintelligence decides to act benevolently.

The same year Metzinger published his essay, an EA named Derek Shiller published an article titled “In Defense of Artificial Replacement.” He argues that,

if it is within our power to provide a significantly better world for future generations at a comparatively small cost to ourselves, we have a strong moral reason to do so. One way of providing a significantly better world may involve replacing our species with something better. It is plausible that in the not-too-distant future, we will be able to create artificially intelligent creatures with whatever physical and psychological traits we choose. Granted this assumption, it is argued that we should engineer our extinction so that our planet’s resources can be devoted to making artificial creatures with better lives.

In this case, Shiller is suggesting that we make the decision to voluntarily die out. But our posthuman descendants might reach the same “moral” conclusion and opt to eliminate us, even if humanity decides it wants to stick around. This alternative possibility is what Metzinger highlights in his essay: precisely because the ASI is fully benevolent and altruistic, it gets rid of us, for the sake of making the world better by removing a population of creatures prone to suffering.

***

If ASI were controllable and to usher in what Yudkowsky calls our “glorious transhumanist future,” there is absolutely no reason to believe that our species would survive. Once again, we’d be using up finite valuable resources and taking up space while being economically useless. Furthermore, an “ethical” posthuman species might simply opt to euthanize us because we’re “too sad” to keep around.

The coexistence view is a complete nonstarter. If we create and/or some portion of humanity becomes a new posthuman species, our days are numbered; terminal extinction will become more or less inevitable. This is why I argue that anyone who accepts a posthuman eschatology, whether they favor replacement or coexistence, is pushing a future in which we will die out. Perhaps in the coming years.

***

It might be tempting to say, “Well, maybe this wouldn’t be such a bad thing, if we were replaced by superior posthumans? Maybe Yudkowsky has a point?” If you agree with this, then you’re a pro-extinctionist. But it’s also worth pointing out that there’s no reason to think that posthuman life would actually be any better. It could, in fact, be far worse in many ways. Here are a few examples:

(1) Imagine undergoing a life-extension intervention that enables you to live at least 10,000 years. You won’t die of old age or disease. The only possible causes of mortality are the same things that kill young people, namely, accidents.

Think about how this would change your quantitative risk assessment of, say, walking through a city. Even if there’s a minuscule chance you might get flattened on the road by a bus, the stakes are so huge — 10,000 future years of life — that the risk would still be enormous.

That’s because “risk” is defined as “the probability of an adverse outcome multiplied by its badness.” This means that a low-probability but very high-stakes outcome could nonetheless yield an extremely high risk. Hence, simply perambulating to the grocery store would become an exceptionally risky affair, which means you should stay home all day and never leave your apartment. That doesn’t sound like a fun life, but it’s the life that one should live if one could survive for a millennium and is thinking rationally.

The same point applies to digital space brains whose physical substrate is computer hardware. If a space brain knows it could potentially survive until the heat death, the stakes become so huge that there’s literally nothing it should do each day for nearly its entire life other than safeguarding its continued existence. Maybe the computer on which it’s running breaks down, and maybe there aren’t enough back-ups of its mind on other servers. Maybe there’s an intergalactic war that blows everything up (see below). Perhaps an asteroid randomly destroys the “planet-sized” computer (quoting Bostrom) that runs the simulation in which it resides.

There are a million possibilities here, and if risk equals probability multiplied by the consequences, then even a minuscule chance of some random event destroying the space brain should occupy it’s thoughts every second of every day.

(2) Now consider the fact that becoming a digital space brain would open up the unspeakably horrifying possibility of being tortured for literally trillions of years. Enemies of the cosmic civilization — political dissidents, criminals, or outcasts — could be locked away in a digital dungeon to suffer excruciating agony every second of the day until the heat death. That’s not something that could happen to us (our species), although radical life-extension technologies of the sort Peter Thiel wants to developed could enable something similar here on Earth until it becomes uninhabitable in 1 billion years.

This is absolutely terrifying, but it could become common in the posthuman era.

(3) There’s also the aforementioned possibility of dictators — perhaps Demis Hassabis, Sam Altman, Donald Trump, or someone else — establishing a totalitarian regime that no one can overthrow. Then, as this regime spreads into space, it could fracture into nations that engage in constant catastrophic wars, as the international relations theorist Daniel Deudney argues in his excellent book Dark Skies. The basic idea is this (paraphrasing a previous newsletter article):

If future beings have the technology to spread beyond our solar system, they will probably also have the technology to inflict catastrophic harm on each others’ galactic neighborhoods. They might employ cosmic weapons we can’t even imagine. Furthermore, outer space will be politically anarchic, meaning that there’s no central Leviathan (or state) to keep the peace. Indeed, since a Leviathan requires timely coordination to be effective, and since outer space is so vast, establishing a Leviathan will be impossible. (In other words, there will be no single civilization, but a vast array of these civilizations, populated by beings with wildly different cognitive capabilities, emotional repertoires, technological capacities, scientific theories, political organizations, and even religious ideologies — etc. etc.)

This leaves the threat of mutually assured destruction, or MAD, as the only mechanism for securing peace. But given the unfathomably large number of civilizations that would exist (because the universe is huge), it would be virtually impossible to keep tabs on all potential attackers and enemies. Civilizations would find themselves in a radically multi-polar Hobbesian trap, whereby even peaceful civilizations would have an incentive to preemptively strike others. Everyone would live in constant fear of annihilation, and the inherent security dilemmas of this predicament would trigger spirals of militarization that would only further destabilize relations in the anarchic realm of the cosmopolitical arena. Meanwhile, those captured from enemy civilizations could be brutalized forever in simulated torture chambers.

This is the stuff of nightmares, but on a cosmic scale, lasting until the heat death.

So, humanity has its downsides, but so will posthumanity. Human beings are constrained by our biological limitations, but posthumans will have their own limitations to contend with. Promises of a “techno-utopia” are nothing more than propaganda to “justify” a race to build ASI that’s leaving a trail of destruction in its wake.

Perhaps humanity, in its current form, is the closest we’ll ever get to paradise, and hence losing humanity would be a catastrophic tragedy.

***

TESCREALists, including those trying to build an AI God, are pushing for a future in which our species will go extinct, perhaps in the coming decades. Because I actually care about avoiding the terminal extinction of our species, I object to anyone who promotes a posthuman eschatology. I think it’s a recipe for disaster. And I don’t think that embracing the coexistence view would save us.

Take a step back for a moment and consider what TESCREALists are offering us. Accelerationists like Beff Jezos offer us one option: the extinction of our species by replacement with posthuman ASI.

Doomers are a bit more generous (!) in offering us two options: the extinction of our species or, alternatively, the extinction of our species. In the first case, we die out because misaligned ASI kills everyone. By virtue of being misaligned, this ASI won’t constitute a “worthy successor.” In the second case, an “aligned” ASI enables some people — the elites — to become posthuman. The remaining humans are then pushed out of existence by those posthumans.

Using my terminology, the first would involve final extinction, while the second would involve terminal without final extinction. In both cases, our species dies out.

This is why I won’t lock arms with TESCREAL doomers who want the ASI race shut down until the alignment problem is solved. They are not on my side; they are not on Team Human. Their allegiance instead is to Team Posthuman — to realizing a posthuman eschatology. That eschatology would almost certainly result in Team Human going extinct.2 This is why Yudkowsky and Beff Jezos occupy the exact same spot in my mind: they are enemies of humanity, our species, and should be dealt with as such.

You might say: If Anyone Becomes Posthuman, Everyone Dies Out.

But what do you think? What have I missed? How might I be wrong? As always:

Thanks for reading and I’ll see you on the other side!

1. Why are you (they say to me) so harshly critical of Eliezer Yudkowsky? Why not join him in warning about the dangers of AGI and calling for an international treaty to stop the AGI race? It’s counterproductive to target someone with such a big public profile! You should be working with rather than against him!

2. Wait, you say that the AGI race should be stopped immediately and that it’s incredibly dangerous, yet you also say that we’re nowhere close to building AGI. How does that make sense?

Taking these in order, a reminder that Yudkowsky has made remarks like these:

If sacrificing all of humanity were the only way, and a reliable way, to get … god-like things out there — superintelligences who still care about each other, who are still aware of the world and having fun — I would ultimately make that trade-off.

Yudkowsky emphasizes that this “isn’t the trade-off we are face with” right now. But if it were, he’d willingly sacrifice our species to see artificial super-beings flitting about the universe “having fun.”

Think for a moment about how utterly outrageous this is. He’ saying that he’d be willing to sacrifice all Chinese people, Indian people, South Africans, Norwegians, everyone in Nigeria, Chile, Mongolia, and Japan, the entire populations of Toronto, Bangkok, Moscow, and London, if doing so were the “only” and a “reliable” way of creating fun-having digital space brains.

That is absolutely abhorrent. It’s atrocious. Imagine saying that you’d be willing to sacrifice my life to realize your dream: a big mansion in Hollywood. Everyone would agree that saying this would be completely unacceptable. Now imagine a TESCREAList, like Yudkowsky, saying they’d sacrifice my life, the lives of my family, and the lives of everyone else on Earth to realize their dream of a cosmic utopia full of digital space brains. I have no words to express how horrendous this is.

My friend Mark Gubrud, who often disagrees with me, wrote this:

(Note that Mark is the person who first coined the term “artificial general intelligence” back in 1997. Shane Legg then seems to have independently coined it in the mid-2000s.)

My response didn’t beat around the bush. “Not sure how else to put this,” I wrote,

there are no doubt many things that 1940s Nazis and I would have agreed about re: the environment. Does that mean I should have locked arms with them in fighting to preserve the natural world? No, absolutely not. Never. You might say this isn’t analogous with the case of Yudkowsky — and I would agree, because what Yudkowsky has said out-loud, in public, is far worse than anything fascists have said. Those fascists were eugenicists who wanted to eliminate specific social groups for the greater good. Yudkowsky is a eugenicist who’s explicitly said that he’d sacrifice literally everyone on Earth to realize his incredibly dumb version of utopia. … What. The. Fuck. What he’s saying here is absolutely fucking insane. It’s among the most extreme, horrific things I’ve ever heard anyone ever say — ever. It’s worse than anything that’s come out of the mouth of Donald Trump, Steve Bannon, or Stephen Miller. Does this make sense now? I want absolutely nothing to do with people who hold fucking insane, incredibly dangers views like those he’s repeatedly expressed.

(Sorry for the vulgarities, but I think they’re warranted in the face of omnicidal threats.)

Here’s a similar example: as you’re all aware, Donald Trump recently said that “a whole civilisation will die tonight, never to be brought back again. I don’t want that to happen, but it probably will.” He then backed off an has apparently reached an agreement according to which Iran gets pretty much everything they wanted (lolz):

The fact that Trump didn’t commit genocidal war crimes against Iran doesn’t for one moment excuse his words. Yet, what Trump said is not as bad as what Yudkowsky has said, and we must acknowledge that. Trump is basically declaring that he’d sacrifice Iran for some obscene notion of the greater good. Yudkowsky is saying he’d sacrifice humanity for some bizarre notion of the greater cosmic good. What is omnicide other than all possible genocides put together?

Furthermore, as I wrote here, Yudkowsky isn’t actually anti-AGI. He’d build AGI literally next week if he thought it was “value-aligned,” meaning aligned with the utopian values of the TESCREAL worldview (cosmic paradise, digital space brains, and all that). He’s not on Team Human, he’s on Team Posthuman.

What I want is to join forces with folks on Team Human who actually do think we should never build AGI. If the alignment problem were solved next week, I’d still oppose AGI. (I provide a detailed argument in my book for why AGI would almost certainly have catastrophic consequences if built, no matter what, which makes me very anti-AGI in an unconditional way.)

***

As for the second question, it might be worth noting that there are at least four main reasons one might think AGI won’t arrive in the near future. These are:

1. The systems that power current AI, large language models (LLMs), aren’t going to get us to AGI by themselves. No matter how much AI companies scale them up by increasing compute (computational resources), training data, and their parameters, they just don’t have the right architecture to become AGI. LLMs by themselves are a dead end, though there might be other systems, architectures, and approaches that could eventually get us there.

2. AGI isn’t a technology that we could ever build. It might simply be too difficult for our species. This is made plausible if one believes there may be other technologies that are in theory possible but in practice out of our reach, such as spacecraft that travel at 99.99% the speed of light. Perhaps some kind of super-clever alien species could figure this out, but we probably never will.

3. AGI, and especially ASI (artificial superintelligence), isn’t possible to build in theory. It’s like a perpetual motion machine, or designing a spacecraft that can exceed the speed of light. There’s just no way to build an artificial system that surpasses human capabilities in every cognitive domain of interest, perhaps because we represent the highest level of “intelligence” attainable.

4. AGI and ASI are not even coherent ideas to begin with. What does it mean to build an “everything machine” that can “exceed” humans in every domain of interest? Heck, AI companies and researchers can’t even agree upon a definition of “AGI” — OpenAI itself proposes multiple inconsistent definitions on its own website! What, then, are we even talking about?

I very much agree with (1), and am sympathetic with (4). However, I also find (2) appealing for the following reason: it could be that a certain degree of societal, political, economic, etc. stability is necessary to build AGI. However, the stepping-stone systems that we’d need to build in order to reach AGI may wreak so much havoc that the fabric of society unravels, thus making AGI unreachable. In other words, there may be a negative feedback loop here such that the closer we get, the further away we end up: building AGI requires societal stability, but the more AI we have, the less stable things become.

On the ineptitude of AI … Shot and chaser:

A chapter in my book goes into immense detail about all the harms caused by current AI systems, which the AI companies have built in hopes of scaling them up to become AGI. When one surveys all these harms catalogued in one place, over the course of 30 pages, it’s completely shocking, and forces one to admit that it’s not implausible the AGI race itself could potentially cause societies to collapse (or at least play a significant causal role in helping along the process).

This is why I call for an immediate, permanent halt to the AGI race — on scaling up generative AI systems. They have very few benefits, and threaten to undermine key civic institutions like the rule of law and free press, as well as facilitating things like domestic mass surveillance.

That’s how someone can simultaneously say that AGI isn’t around the corner and that the race to build AGI must stop immediately. To this, I’d also add that if we were to someday build AGI, I have no reason to expect it to be controllable by default. If humanity were suddenly joined by an entity that could outmaneuver us in every important respect, solve complex problems faster than any human, set and modify its own goals without human intervention, and act as an autonomous agent in the world, etc., then we’re in very big trouble.

Fortunately, we’re nowhere close to building such an entity. Unfortunately, we don’t need AGI for AI to destroy the world.

***

Speaking of my book, I now have 85,000 words written! I’m almost done, which means I’ll get back to posting two articles per week in probably less than a month. The book will end up around 100,000 words long, and my hope is that it offers a devastating and original critique of the AGI race (I don’t know of anyone else who makes the points I do).

But I’m in a bit of a pickle, as I’m struggling to find a new publisher who can get it out later this year — rather than 13 months from now, which is what my current publisher is offering. Who the hell knows what the world will look like in 13 months (lol); the topics I discuss are relevant right now!

I have a few promising leads, and I’m not fundamentally opposed to self-publishing it in the end if that’s necessary to get it out sometime in 2026. (The book will still be massively peer reviewed either way, as I send most of what I write out to many dozens of people for feedback and criticisms.) Anyways, just giving you an update. The current table of contents is:

Clown Car Utopia: Why We Must Stop AI to Save Humanity

• Chapter 1: Digital Space Brains

• Chapter 2: An Alphabet Soup of Ideologies

• Chapter 3: The Road to Utopia

• Chapter 4: Death to Humanity

• Chapter 5: A Trail of Destruction

• Chapter 6: Our Precious Planet

• Chapter 7: Butlerian Jihadists

I honestly could not do this without your support! As always:

Thanks for reading and I’ll see you on the other side!

That can’t be said about a review of the book published in The Sydney Morning Herald. The author, Pat Sheil, repeatedly describes as a TESCREAList. He writes:

Emile P. Torres is a long-term mover and shaker in the eternal life caper. And there are many rich people giving money to people like them.

It’s not true that “many rich people” in the longevity community are funding my work, because no one in or around the TESCREAL movement is funding me. (Obviously!!) He continues:

And here’s where it gets scary, and why Krotoski’s book is important. It turns out that Torres remains an influential theorist and strategist for immortalists, and as such a most convincing preacher to insatiable venture capitalists (most of whom wouldn’t mind living forever either, funnily enough). Hence the signing of so many fat cheques.

Torres believes that anyone who stands in the way of this project is, by definition, a threat to the human species, and should logically be prevented from getting in their way. Their project includes mind-merging with AGI (Artificial General Intelligence, or the Singularity; the much-hyped “all-wise, all-knowing” son of AI), which thankfully doesn’t yet exist, and hopefully never will.

LOL and WTF.

This is defamatory, almost as bad as that Guardian article from a few years ago that misquotes me to suggest I’m a pro-extinctionist (I’m not).

I wrote the Herald, asking them to retract the article or at least correct the record, but they have yet to respond. Though I’m a little miffed about this, I guffawed while reading: it’s such a gross misrepresentation of my view, and the account of my view in The Immortalists, that I wonder if ChatGPT wrote the entire article? Genuinely quite amusing, amateurish stuff.

The 21st-Century Existential Mood

In my 2024 book Human Extinction, I argued that Western history can be divided into five distinct periods, each defined by a specific set of answers to questions like: Is human extinction possible? If so, how could it come about? What is the probability of our extinction? Is it inevitable in the long run? Could it happen in the near future? And so on.

I found it astonishing how abrupt the shifts from one period to another were. Over the course of a single year, or at most a decade, previously established answers to these questions were dramatically overthrown, often inducing a degree of psycho-cultural trauma. For example, virtually no one was talking about human extinction in the years just after 1945. Then the Castle Bravo disaster happened in 1954, which involved a thermonuclear weapon being detonated in the Marshall Islands and catapulting radioactive particles around the entire globe. Almost overnight, a very large number of eminent scientists began declaring that even a small-scale thermonuclear war could render Earth completely unsuitable for human life, thus resulting in “universal death.”

Another shift happened in the early 1850s, when scientists discovered the second law of thermodynamics. Over the course of just a couple years, people went from saying that human extinction probably isn’t even possible to acknowledging that it’s inevitable, as our sun burns out and Earth becomes an icy tomb floating about the darkness.

I argue that each of these periods correspond to a unique “existential mood,” by which I mean a kind of public mood — as in, the “mood” of the 60s was one of rebellion. Bertrand Russell captures the essence of the mood that arose after the second law was discovered, writing in his lugubrious 1903 “A Free Man’s Worship”:

All the labours of the ages, all the devotion, all the inspiration, all the noonday brightness of human genius, are destined to extinction in the vast death of the solar system, and that the whole temple of Man’s achievement must inevitably be buried beneath the debris of a universe in ruins.

I mention this because I argue in the book that our current existential mood was initiated in the late 1990s and early 2000s, and is marked by the sense that, however perilous the 20th century was, the worst is yet to come. The 21st century will be even more dangerous than the 20th. This is for two reasons:

First, a consensus emerged in the early 2000s that climate change is both anthropogenic and could have catastrophic consequences. (Yes, people had been talking about climate change for decades, but the issue was debated among scientists until the early aughts, when virtually all climatologists came to agree about its underlying causes and likely effects.)

Second, new anxieties about the immense destructive power of emerging technologies, including biotechnology and artificial intelligence. Bill Joy’s widely discussed 2000 Wired article, “Why the Future Doesn’t Need Us,” exemplified these anxieties. He argued that emerging tech like AI could be so dangerous that we should impose broad moratoria on entire fields of emerging science and technology. That’s basically what people like Eliezer Yudkowsky and groups like Stop AI are arguing right now.

I have watched, in realtime, this mood spread across the Western world. There’s a difference, I argued, between when a new mood first emerges and when it becomes widespread within a society. The existential mood that emerged around the turn of the century is now in full bloom. It’s everywhere you look, and many of us can feel it in our bones. I can’t go a single day without seeing dozens, if not hundreds, of posts on social media claiming that AI could destroy humanity in the coming years. Meanwhile, news about the climate crisis continues to cast a dark shadow over civilization, fueled by new research showing that global warming appears to be accelerating. Something terrible is about to happen — that’s the essence of our current mood, and expressions of it and the psycho-cultural trauma it’s inflicting are increasingly omnipresent.

In a bit more detail:

There’s growing talk about a nuclear weapon being used against Iran, which doesn’t seem out of the question given that Israel just committed a genocide (indicating that it cares not about violating international laws, norms, and taboos), and the US is run by a demented madman who seems hangry for geopolitical conflict (Venezuela, Iran, annexing Greenland, talk of taking over Cuba, etc.).

With respect to climate change, a recent study found that “climate change’s rising seas may threaten tens of millions more people than scientists and government planners originally thought because of mistaken research assumptions on how high coastal waters already are.” In a Nature article titled “The World Just Lived Through the 11 hottest Years on Record — What Now?,” the authors write that “measurements of Earth’s energy input and output reveals that the planet is more out of balance than ever before.” It includes this quote:

“We seem to be entering this new era where temperatures will be significantly higher than what they were ten years ago,” says climate scientist Sarah Perkins-Kirkpatrick at the Australian National University in Canberra. The past three years have seen large changes in temperature that could only be a result of climate change, she adds.

Another study in Nature finds that

extreme global climate outcomes may occur even under moderate 2 °C warming for several sectors. For droughts in global key breadbasket regions, precipitation extremes over highly populated areas and fire weather extremes across forests, global climatic impact-drivers at 2 °C of global warming may turn out to be much more extreme than model-averaged projections at 3 °C or 4 °C warming.

We’re already at 1.5C of warming — 2024, the hottest year on record, reached about 1.55C above pre-industrial levels. 2023 was the second hottest, and 2025 the third. Yet this year could exceed the record, and indeed studies suggest there may be a “globally catastrophic” super-El Nino event forming by spring. This could make 2027 even worse than 2026, as each of the three super-El Nino events since 1980 have been “followed by a year of record-breaking heat globally.”

Already this year, sea-surface temperatures are off the charts (below), and “new evidence shows antarctic melting is already locked in,” meaning that there’s nothing we can do at this point to avoid devastating sea-level rise, which will affect upwards of 1 billion people.

Just this month, temperature records were broken throughout the US, likely setting an all-time record for the month. As CNN reports, the city of Yuma, Arizona, saw temperatures soaring to 109F, while “the temperature near Martinez Lake, Arizona, hit 110 degrees on Thursday and 112 degrees on Friday” (the 19th and 20th of March). To quote the CBC:

A huge heat dome is spreading across the United States and it is shattering March temperature records. Weather historians say the dome has already smashed statewide March records in 14 states. Now, the gigantic heat dome that’s baked the Southwest is creeping eastward and may end up being one of the most expansive heat waves in American history, meteorologists and weather historians said. Experts say the heat wave’s footprint may rival major events in 2012 and 2021.

Another study reports that the US’s carbon emissions may have caused $10 trillion in damage since 1990. Much of this, of course, disproportionately hurts the most vulnerable people around the world who, historically, have contributed the least to climate change. That’s the main focus of climate justice.

Adding to the insanity of this situation, Trump just declared environmentalists to be “terrorists.” I guess that makes me a “terrorist”? For, you know, wanting humanity to not destroy our exquisitely unique, beautiful little oasis in space? What a joke.

That said, almost no one thinks that climate change will cause our extinction — the complete elimination of every person on Earth. But it could very well push civilization over the precipice of collapse.

Recall a University of Exeter study that calculates a GDP loss of greater than or equal to 25% and over 2 billion deaths if we reach 2C of warming by 2050. If we reach 3C, we should expect a greater than or equal to 50% loss of GDP and more than 4 billion deaths. This is an incredibly dire situation, yet climate apocalypticism has been largely eclipsed in the popular media by warnings that an omnicidal AGI might kill humanity before climate change topples civilization. It has been incredible to see this idea metamorphose from an obscure worry held by a fringe group of AI doomers into a topic now discussed on major media outlets like CNN.

In the next few days, I’m hoping to see The AI Doc, which has received quite a bit of attention. As far as I can tell, it’s mostly about the internecine squabbles between people within the TESCREAL movement — e.g., the doomers versus the accelerationists. The former believe that AI capabilities research should be stopped until AI safety researchers have solved the control problem, whereas the latter don’t seem to care one bit if ASI annihilates humanity. As the computer scientist David Krueger writes, “there are a significant number of people in the AI research community who explicitly think humans should be replaced by AI.” To which Max Tegmark replied: “I’ve been shocked to discover exactly this over the years through personal conversations. It helps explain why some AI researchers aren’t more bothered by human extinction risk: It’s not that they find it unlikely, but that they welcome it.”

I will almost certainly write a review of the film, though that might be difficult given that I’m completely immersed in writing my book. Currently halfway through chapter 5, after which I’ll have only two chapters left. I cannot wait for this to get published, because I think (hope) it offers a devastating and original critique of the ASI race — one that no one else is making! Thank you so much for supporting me while I work on this project, by the way!!

Incidentally, I have a section in the book in which I discuss the reasons one might have for rejecting the claim that ASI might be imminent, and that once here it will annihilate humanity by default. I’m very curious about your thoughts — am I missing something? What reasons do you have for rejecting TESCREAL doomerism? Here’s what I write:

… In fact, I would go further and argue that we should all be outraged even if one thinks an ASI-induced extinction catastrophe won’t happen in the near future — or ever. You might believe, for example, that

1. The systems that power current AI, large language models (LLMs), aren’t going to get us to AGI by themselves. No matter how much AI companies scale them up by increasing compute (computational resources), training data, and their parameters, they just don’t have the right architecture to become AGI. LLMs by themselves are a dead end, though there might be other systems, architectures, and approaches that could eventually get us there.

2. AGI isn’t a technology that we could ever build. It might simply be too difficult for our species. This is made plausible if one believes there may be other technologies that are in theory possible but in practice out of our reach, such as spacecraft that travel at 99.99% the speed of light. Perhaps some kind of super-clever alien species could figure this out, but we probably never will.

3. AGI, and especially superintelligence, isn’t possible to build in theory. It’s like a perpetual motion machine, or designing a spacecraft that can exceed the speed of light. There’s just no way to build an artificial system that surpasses human capabilities in every cognitive domain of interest.

4. AGI and ASI are not even coherent ideas to begin with. What does it mean to build an “everything machine” that can “exceed” humans in every domain of interest? Heck, AI companies and researchers can’t even agree upon a definition of “AGI” — OpenAI itself proposes multiple inconsistent definitions on their own website! What, then, are we even talking about?

I am very sympathetic with the first view: LLMs are not a ticket to AGI, though I’m also sympathetic with claims that “AGI” might not be a coherent concept in the first place. Insofar as it is coherent, I suspect it might not be possible for us to build, but for a different reason than stated above: it could be that a certain degree of societal, political, economic, etc. stability is necessary to build AGI. However, the stepping-stone systems that we’d need to build in order to reach AGI may wreak so much havoc that the fabric of society unravels, thus making AGI unreachable. In other words, there may be a negative feedback loop here such that the closer we get, the further away we end up: building AGI requires societal stability, but the more AI we have, the less stable things become. We’ll return to this in a moment.

What do you think? Here’s a poll, but I’d also love to know your thoughts in the comments section.

As always:

Thanks for reading and I’ll see you on the other side!

I’m worried that most people don’t understand who Yudkowsky is and what he actually believes. (He’s literally argued that a child only has a “right to live” “sometime after 1 year and before 6 years.”) Here’s a short excerpt from my forthcoming book, Clown Car Utopia: Why We Must Stop AI to Save Humanity:

Another figure who’s repeatedly expressed pro-extinctionist views is Yudkowsky himself. This may be surprising given that Yudkowsky frequently talks about the importance of avoiding human extinction, but we’ll see below that what he means by “human extinction” is very different from what the rest of us mean. In fact, he advocates for a future in which our species will almost certainly die out by being replaced by posthumanity — ideally, the particular version of posthumanity that he favors, which is not the same posthumanity as what accelerationists like Larry Page, Richard Sutton, and Gill Verdon (“Beff Jezos”) endorse.

During a conversation with explicit pro-extinctionist Daniel Faggella on The Trajectory podcast, Yudkowsky declared that

if sacrificing all of humanity were the only way, and a reliable way, to get … god-like things out there—superintelligences who still care about each other, who are still aware of the world and having fun—I would ultimately make that trade-off.

Yudkowsky emphasizes that this “isn’t the trade-off we are face with” right now. But if it were, he’d willingly sacrifice our species to see artificial super-beings flitting about the universe “having fun.” He repeated this idea during a recorded conversation with Stephen Wolfram, a computer scientist who delivered talks at the Singularity Summit in 2009 and 2011 (cofounded by Yudkowsky, Ray Kurzweil, and Peter Thiel). “It’s not that I’m concerned about being replaced by a better organism,” he told Wolfram, “I’m concerned that the organism wouldn’t be better.” Once more, replacement itself isn’t the issue.

Yudkowsky went into even more detail on the Bankless podcast in 2023, arguing that once creating posthumanity becomes feasible, it may be unethical to have biological children. Using rather offensive language, he said:

I have basic moral questions about whether it’s ethical for humans to have human children, if having transhuman children is an option instead. Like, these humans running around? Are they, like, the current humans who wanted eternal youth but, like, not the brain upgrades? Because I do see the case for letting an existing person choose “No, I just want eternal youth and no brain upgrades, thank you.” But then if you’re deliberately having the equivalent of a very crippled child when you could just as easily have a not crippled child.

Yudkowsky continued:

Like, should humans in their present form be around together? Are we, like, kind of too sad in some ways? I have friends, to be clear, who disagree with me so much about this point. (laughs) But yeah, I’d say that the happy future looks like beings of light having lots of fun in a nicely connected computing fabric powered by the Sun, if we haven’t taken the sun apart yet. Maybe there’s enough real sentiment in people that you just, like, clear all the humans off the Earth and leave the entire place as a park. And even, like, maintain the Sun, so that the Earth is still a park even after the Sun would have ordinarily swollen up or dimmed down.

Okay, so. Get rid of humanity and turn Earth into a nature reserve. Meanwhile, posthumans would reside in virtual-reality worlds (what he calls “computing fabric”) powered by megastructures called Dyson swarms that envelope the Sun and harvest nearly all of its energy output.

Kokotajlo, another TESCREAL utopian, has made similar remarks, as when he told the New York Times that

I’m a huge fan of expanding into space. I think that would be a great idea. And in general, also solving all the world’s problems, like poverty and disease and torture and wars. I think if we get through the initial phase with superintelligence, then obviously, the first thing to do is to solve all those problems and make some sort of utopia, and then to bring that utopia to the stars would be the thing to do.

The thing is that it would be the AIs doing it, not us. In terms of actually doing the designing and the planning and the strategizing and so forth, we would only be messing things up if we tried to do it ourselves.

So you could say it’s still humanity in some sense doing all those things, but it’s important to note that it’s more like the AIs are doing it, and they’re doing it because the humans told them to (italics added).

I suspect that if you knew something about the TESCREAL movement to which these people belong, you would be rather mortified. They are not actually opposed to superintelligence, nor are they pro-human, as I explain in detail here.

To the contrary, Yudkowsky’s institute, MIRI, explicitly says:

We remain committed to the idea that failing to build smarter-than-human systems someday would be tragic and would squander a great deal of potential. We want humanity to build those systems, but only once we know how to do so safely.

The whole point of building a “value-aligned” superintelligence, they say, is to be aligned with the values of TESCREAL utopianism — to radically transform us into digital posthumans, and then colonize space and conquer the universe. This is not an exaggeration. It’s the core vision of the TESCREAL ideologies.

I am begging you to please stop joining hands with pro-extinctionists. Yes, their interests are temporarily aligned with yours: shut it all down, a position that I myself passionately advocate. But siding with them is like siding with a murderer who says he won’t kill you until next year. I get the impression that you are on Team Human, on the side of our species. They are not. They are on Team Posthuman.

Another thing: you’ll hear people like Yudkowsky talk about the importance of avoiding “human extinction.” But what he means by the term is not what you think. People in the TESCREAL movement define “human” in an idiosyncratic manner — to mean our species plus whatever posthuman successors we might create. On this definition, our species could die out next year without human extinction having happened. So long as we have posthuman successors to take our place, then “humanity” will persist, and hence human extinction will not have occurred.

Here are some examples from TESCREALists in the same tradition as Yudkowsky (quoting from a peer-reviewed article of mine):

Nick Beckstead (2013) writes that “by ‘humanity’ and ‘our descendants’ I don’t just mean the species homo sapiens [sic]. I mean to include any valuable successors we might have,” which he later describes as “sentient beings that matter” in a moral sense. Hilary Greaves and MacAskill (2021) report that “we will use ‘human’ to refer both to Homo sapiens and to whatever descendants with at least comparable moral status we may have, even if those descendants are a different species, and even if they are non-biological.” And Toby Ord (2020) says that “if we somehow give rise to new kinds of moral agents in the future, the term ‘humanity’ in my definition should be taken to include them.”

Please don’t be fooled by their linguistic trickery. When they talk about preventing human extinction, they aren’t talking about ensuring the survival of our species. Our survival only matters insofar as it’s necessary to bring about what Yudkowsky calls the “glorious transhumanist future.”

Perhaps I have misread your positions. Bernie, Neil, and anti-AI protestors may very well be on Team Posthuman. I hope that’s not the case. If you are indeed on Team Human, please stop propping up people who say they’d literally “sacrific[e] all of humanity” to create “worth successors” in the form of artificial superintelligence.

Sincerely, Émile

Again, this is the system that Reid Hoffman (friend of Epstein!) called “Universal Basic Superintelligence,” and which Sam Altman touted last year as having PhD-level knowledge. In truth, it’s nothing more than a stochastic parrot vomiting up bits and pieces of its training data, leading to egregious failures of basic reasoning like the one above.

Also, I’m sure that many of you have seen this by now, but in case you haven’t, here’s Marc Andreessen — noted AI accelerationist who once included “TESCREAList” in his Twitter bio — claiming that he never engages in any introspection:

He then said “I regret nothing” after his remarks went viral for all the wrong reasons. To that, I responded:

Silicon Valley truly is run by sociopaths with no empathy for others and no ability or desire to reflect on their behaviors, feelings, and ideas. Which leads us to the main topics of this post!

The Race to Superintelligence Isn’t a Collective Action Problem

I’m now 40,000 words into my book — just finished chapter 3. Thank you so much for supporting me while I write this. It might be done in another two or three weeks!

The working title is Clown Car Utopia: Why We Must Stop AI to Save Humanity, though I argue that we can’t stop AI without stopping, countering, defanging, and neutralizing the TESCREAL movement. This will be the culmination of four years of academic and popular media articles, as well as countless podcast, radio, and TV interviews that I’ve done. But really, it’s the culmination of 20 years of research, as I first became interested in the TESCREAL movement around 2006, after stumbling upon the work of Ray Kurzweil and Nick Bostrom. Every single page of this book contains something that will have readers saying, “What the hell did I just read?” — because the TESCREAL movement is endlessly bizarre, outrageous, and absurd. In a sense, the book contains a “greatest hits” catalogue of the most cockamamie things TESCREALists have said and done, with receipts.

While researching a section of the book, I re-listened to an interview with Holden Karnofsky on the 80,000 Hours podcast. Karnofsky was an important figure in the early development of EA. He started GiveWell, which worked closely with Toby Ord and William MacAskill’s Giving What We Can, and cofounded Open Philanthropy (now Coefficient Giving). He was roommates with Dario Amodei, the CEO of Anthropic, and married Dario’s sister Daniela. He’s now a member of the technical staff at Anthropic, where he advises “the company on preparing for risks from advanced AI.”

In the interview, he says that “the AGI race isn’t a coordination failure.” That it is a coordination failure is the standard story told — one finds it, for example, in Karen Hao’s excellent book Empire of AI. The story goes:

Every leading figure at the major AI companies — DeepMind, OpenAI, Anthropic, and xAI — believes that AI safety is extremely important. If we build a “value-misaligned” ASI (artificial superintelligence), it will destroy humanity along with our “glorious transhumanist future” among the stars. But if we ensure that it embodies “our values” (by which they mean the values of the TESCREAL worldview), then we get a posthuman paradise, a heaven among the literal heavens.

The problem is that each company thinks that it’s more responsible than the others, and hence that it should be the one to reach the ASI finish line before everyone else. No one wants a race — they just look around at the other companies and conclude, “Well, if we don’t speed up, they’re going to get to ASI before us, which means the probability of doom will be higher than if we got there first.” Hence, the coordination or collective action problem driving the arms race.

Karnofsky disagrees — and I agree with his assessment. This isn’t a collective action problem. Why? Because many people at the AI companies (a) don’t care if ASI wipes out humanity; some think that would actually be a good thing, or (b) think the risk of annihilation is completely worth it for the chance that they get to become immortal posthumans.

This is exactly the line of reasoning in Bostrom’s atrocious new paper arguing for the accelerationist thesis: if ASI is sufficiently value-aligned to give us more than a thousand years of extra life, then we should push ahead even if the probability of ASI being value-aligned is only 3%. In other words, we should build ASI as soon as possible even if the probability of doom is 97%. If you don’t believe him, just do the math: the expected value of risking almost certain annihilation is higher than the alternatives if ASI lengthens our lifespans by more than 1,000 years. As Eliezer Yudkowsky says, “Shut up and multiply!”

The reasoning here, by the way, is called “Pascal’s mugging.” It’s a terrible form of argument, but one that many people at these AI companies embrace. Put more colloquially, these people are gripped by a kind of YOLO (you only live once) attitude: “Look, if I’m going to die someday anyways, why not race ahead in hopes of living forever, even if doing this risks the lives of everyone on Earth? It’s now or never, baby, so pedal to the metal!”

Here’s what Karnofsky says:

I think most of the players in AI are going to race. And if, for example, Anthropic were to say, “We’re out. We’re going to slow down,” they would say, “This is awesome. That’s the best news. Now we have a better chance of winning, and this is even good for our recruiting” — because they have a better chance of getting people who want to be on the frontier and want to win.

When asked whether OpenAI, DeepMind, and xAI would slow down if Anthropic dropped out, he said:

Let’s take an even stronger hypothetical. Let’s say that not only Anthropic, but everyone in the world who thinks roughly the way I do — everyone in the world who thinks AI is super dangerous, and it would be ideal if the world would move a lot slower, which I do think — let’s say that everyone in the world who thinks that decided to just get nowhere near an AI company, nowhere near AI capabilities. I expect the result would be a slight slowing down, but not a large slowing down.

I think there’s just plenty of players now who want to win, and they are not thinking the way we are, and they will snap up all the investment and capital and a lot of the talent.

During an interview I gave last week, I explained that there are two general groups of accelerationists. The first group thinks that ASI will by default be value-aligned. Marc Andreessen seems to hold this view. He appears to think that if we just plow ahead with ASI, it will by default bring about a utopian world of radical abundance, human enhancement, and space colonization. The second group thinks that it doesn’t even matter whether ASI is value-aligned. Indeed, many argue that ASI shouldn’t be value-aligned — it should have its own “alien, inhuman” values.

In both cases, the conclusion is — as noted — pedal to the metal. That’s why Karnofsky thinks this isn’t a coordination problem. Even if one, two, or all of the companies disbanded, there would still be people who’d immediately start new companies to race toward ASI as quickly as possible.

There’s no stopping the ASI race unless the government swoops in and imposes robust regulations to prevent this from happening. And since the government won’t do that, we’re kinda screwed — not because ASI is actually around the corner. I don’t believe that at all. But rather, because the race itself is causing profound harms to the world. Because we don’t need AGI or ASI for AI to destroy our society.

How in the Hell Is This Acceptable?

That brings me to the second issue, which I’ve written about before. If you were to send me a death threat, you might get arrested and charged. If you were to say, “Okay, I might not actually kill you, but there’s a real chance that I will,” you could still get in trouble. I’d call the authorities and they’d act accordingly.

However, if you say, “I might kill everyone on Earth,” you apparently won’t get in any trouble at all. No one will call the cops, the authorities won’t act, and nothing would ultimately happen. I know this because virtually all the AI company CEOs or founders have said something exactly like that: “We’re building a technology that might kill you, your partner, your mother and father, your children and grandchildren (if you have any), your grandparents (if they’re still around), and all your friends in the near future.”

Sam Altman (CEO of OpenAI) says that

• “the bad case ... is lights out for all of us.”

• “machine intelligence is something we should be afraid of.”

• “AI will … most likely sort of lead to the end of the world, but in the meantime there will be great companies created with serious machine learning.”

• “probably AI will kill us all, but until then we’re going to turn out a lot of great students.”

Dario Amodei (CEO of Anthropic) says there’s a 25% chance of total annihilation. He claims that

• “there’s a long tail of things of varying degrees of badness that could happen. … I think at the extreme end is the … fear that an AGI could destroy humanity. I can’t see any reason in principle why that couldn’t happen.”

Demis Hassabis (CEO of DeepMind) says that

• the probability of total annihilation is “definitely non-zero and it’s probably non-negligible. So that in itself is pretty sobering.”

• we must “take the risks of AI as seriously as other major global challenges, like climate change. … It took the international community too long to coordinate an effective global response to this, and we’re living with the consequences of that now. We can’t afford the same delay with AI.”

Shane Legg (cofounder of DeepMind) puts the probability of extinction from ASI between 5% and 50%, and writes that

• “a lack of concrete AGI projects is not what worries me, it’s the lack of concrete plans on how to keep these safe that worries me.”

• “eventually, I think human extinction will probably occur, and technology will likely play a part in this.”

Elon Musk (cofounder of OpenAI and xAI) argues that ASI poses the “biggest existential threat” to humanity, and claims that it’s “potentially more dangerous than nukes.” He adds that

• “with artificial intelligence, we are summoning the demon. You know all those stories where there’s the guy with the pentagram and the holy water and he’s like, yeah, he’s sure he can control the demon? Doesn’t work out.”

The point I want to get across is this: even if you think these people are crazy for thinking ASI might be imminent and could potentially annihilate humanity, it’s absolutely f-ing outrageous that they get away with saying stuff like this.

In a sane world, they would be locked up and their companies dissolved. In a sane world, saying that you might do something that kills everyone on Earth would be worse than saying you might do something that kills “only” one or two people. Why is it unacceptable to suggest you might murder one or two people but somehow okay to suggest you might kill 8.2 billion?

Even more, what kind of profoundly unethical sociopath even suggests they might kill 8.2 billion people in the first place? Imagine your best friend or partner sitting you down one evening at the dinner table and telling you with a straight face: “I’m going to do something that might kill everyone on Earth, including you.” You joke, “Pfff, what are you talking about? Is this supposed to be a joke?” They say, “No, it’s not. I’m 100% dead serious.”

You would, of course, be completely freaked out by that. It’s a really, really weird thing for someone to say out-loud. You might wonder if they’re having a mental breakdown, or experiencing an episode of psychosis. Because what they’ve just said to you isn’t sane, normal, or acceptable. If, over the next several weeks, they continue to repeat this claim, you might even end your friendship with them — perhaps after calling the authorities or a mental health specialist.

Yet AI leaders have repeatedly said just this, in public. I don’t understand how they aren’t constantly deluged on social media with posts from people saying “F*ck you for threatening to kill me and my family.”

Again, one doesn’t need to believe that an ASI apocalypse is imminent to hold this attitude — it’s an incredibly disturbing thing to hear out of anyone’s mouth, especially the mouths of billionaires running companies with valuations of hundreds of billions of dollars. We do not live in a sane world.

I am personally very angry that Altman and the others have said they might kill me and my family, even though I don’t think we’re anywhere close to ASI. Totally unacceptable behavior.

But what do you think? Am I somehow wrong? As always:

Thanks for reading and I’ll see you on the other side (next week)!

#cancelGPT has been trending since OpenAI took the deal with the Department of War that Anthropic turned down. Tech Crunch reports that “ChatGPT uninstalls surged by 295%” after this happened. For a brief moment, Dario Amodei, the CEO of Anthropic, looked like a hero. But this was quickly followed by a tsunami of negative coverage on social media: Why was Amodei working with the US government, currently run by a fascist regime, in the first place?

Dario is not the good guy, and in fact Anthropic’s Claude was used “to plan the attack on Iran, in the days after [the US military] went to war with the company.” Claude was also used in the illegal kidnapping of Maduro, the president of Venezuela.

Tyler Harper of The Atlantic wondered whether Claude may have been responsible for the US bombing a girls school in Iran, resulting in up to 180 children being brutally murdered. This is entirely possible, given that, as Gary Marcus notes in an article titled “Is AI already killing people by accident?,” “generative AI continues to have serious problems with reasoning and with visual cognition.”

It also turns out that “Anthropic was among the AI companies that submitted a proposal earlier this year to compete in a $100 million Pentagon prize challenge to produce technology for voice-controlled, autonomous drone swarming, acc to people familiar w/ matter.”

A document from Anthropic also circulated in which the company affirms that “Anthropic has much more in common with the Department of War than we have differences.” They’re trying to repair their relationship with a fascist-run military. Amodei has further said, explicitly, that he’s not opposed to his AI systems controlling lethal autonomous weapons (LAWs), i.e., systems capable of choosing, identifying, and killing targets without any human intervention.

This is absolutely outrageous. If an AI-controlled LAW kills an innocent civilian, who does one hold responsible for this? Sentencing a LAW to 10 years in prison wouldn’t bring justice; it makes no sense to punish large language models! Apparently, this is part of the appeal of such technologies for people like Pete Hegseth: no one can be held accountable for charred babies in the street.

A recent study from King’s College London found that when AIs — ChatGPT, Claude, and Gemini — are included in simulated geopolitical crises, they exhibit a distinct tendency to launch nuclear weapons. As New Scientist puts it:

Kenneth Payne at King’s College London set three leading large language models – GPT-5.2, Claude Sonnet 4 and Gemini 3 Flash – against each other in simulated war games. The scenarios involved intense international standoffs, including border disputes, competition for scarce resources and existential threats to regime survival.

The AIs were given an escalation ladder, allowing them to choose actions ranging from diplomatic protests and complete surrender to full strategic nuclear war. The AI models played 21 games, taking 329 turns in total, and produced around 780,000 words describing the reasoning behind their decisions.

In 95 per cent of the simulated games, at least one tactical nuclear weapon was deployed by the AI models. “The nuclear taboo doesn’t seem to be as powerful for machines [as] for humans,” says Payne.

Is this how the world ends? The Department of War, run by a former Fox News host, delegates critical war decisions to AIs that gleefully opt to launch a nuclear strike?

As it happens, Anthropic also reneged on a safety pledge once held up as evidence that it’s more ethically responsible than other AI companies. As TIME reports:

Anthropic, the wildly successful AI company that has cast itself as the most safety-conscious of the top research labs, is dropping the central pledge of its flagship safety policy, company officials tell TIME.

In 2023, Anthropic committed to never train an AI system unless it could guarantee in advance that the company’s safety measures were adequate. For years, its leaders touted that promise — the central pillar of their Responsible Scaling Policy (RSP) — as evidence that they are a responsible company that would withstand market incentives to rush to develop a potentially dangerous technology.

Apparently, the hardcore EA-longtermist Holden Karnofsky had something to do with this decision.

Anthropic’s chief science officer, Jared Kaplan, explained the decision to TIME:

We felt that it wouldn’t actually help anyone for us to stop training AI models. … We didn’t really feel, with the rapid advance of AI, that it made sense for us to make unilateral commitments … if competitors are blazing ahead.

As Holly Elmore, who has emerged (in my mind, at least) as a clear voice of moral sanity in the AI debate right now, puts it without mincing words:

The fact that Anthropic’s plan was to limit contracted use of Claude is so fucking irresponsible I can’t even. They made the mass surveillance murderbot machine and now they want to act shocked that their clients or competitors or distillation hackers are going to use it.

They made their call when they raced to build scaling AI. They don’t have more control than that, and they fucking knew that. Their hope is that the worst loss-control-scenarios wouldn’t come to pass they would somehow end up on top of the economic and geopolitical chaos they created, like by being important to NatSec. Anthropic are absolute villains who have played with your lives since their inception. The point was to coup the world.

Meanwhile, Dario Amodei is out there claiming that “the company is no longer sure Claude isn’t conscious.” Look, this is an incredibly complicated and abstruse issue. There are plenty of reputable contemporary philosophers who are panpsychists, meaning they believe that literally everything has some degree of consciousness, even atoms.

I myself have no idea if artificial systems could be conscious — maybe they can, and maybe LLMs instantiate the right kind of functional organization to give rise to subjective experiences. But if so, Dario, then why in the hell are you building such systems? Why not stop right now and spend the rest of your career calling out the race to build AI super-beings?

A reminder: Claude and all the other AI models are based on massive amounts of intellectual property theft. Anthropic even paid out $1.5 billion in damages for having illegally downloaded copyrighted material from shadow libraries like LibGen. In every way, even the “most ethical” AI company out there has left a trail of destruction behind it.

And for what? The ruination of the Internet, given the slop that now saturates it? As a recent study found, AI poses a direct, immediate, and dire threat to civic institutions like the rule of law, the free press, and universities. I highly recommend this excellent article, titled “How AI Destroys Institutions.” Other studies have found that unrestricted ChatGPT use among undergraduates impairs “long-term retention” of knowledge, “likely by reducing the cognitive effort that supports durable memory.” The paper continues:

The findings align with cognitive offloading theory and the “desirable difficulties” principle: while AI assistance may ease initial learning, it appears to undermine the effortful processes needed for robust learning.

Another “used electroencephalography (EEG) to assess cognitive load during essay writing,” and found “significant differences in brain connectivity” between those who used only their brains (“brain-only”), a search engine to help them, and an AI model based on an LLM. “Brain-only participants,” the report states, “exhibited the strongest, most distributed networks; Search Engine users showed moderate engagement; and LLM users displayed the weakest connectivity. Cognitive activity scaled down in relation to external tool use.” In other words, “ChatGPT users showed 55% weaker brain connectivity than people who didn’t use it. Not after years. After just four months,” with a single session happening each month.

Yet another study reports that, “while LLMs achieve 84 to 89% correctness on synthetic benchmarks, they attain only 25 to 34% on real-world class tasks.” Once in the real world, LLMs that scored very high on benchmarks suddenly aren’t so reliable. Perhaps this is why MIT found that 95% of AI pilots at companies are failing.

An even more recent study “published by the National Bureau of Economic Research” finds that “around 90 percent of the nearly 6,000 interviewed CEOs, chief financial officers, and other top executives at firms across the US, UK, Germany, and Australia, said that AI has had no impact on productivity or employment at their business.”

Some workers report that AI boosts their productivity, but only by creating “workslop” that’s passed on to others, ultimately reducing overall productivity.

AI poses a rapidly growing threat to the world. AI CEOs “justify” this by claiming that Claude, ChatGPT, Gemini, and xAI are the stepping stones to something much greater: AGI, which will trigger the Singularity and lead to a world of endless abundance and unfathomable awesomeness.

They are lying, or delusional. LLMs are a dead end, because problems like hallucinations and their inability to form a coherent world model are inherent in their architecture. The only imminent Singularity is one in which the Internet becomes flooded with AI slop that destroys civic institutions and makes everyone a little dumber.

AI can’t even reliably perform simple tasks right now, for goodness sake, as this person points out:

I built a ClawdBot a couple of days ago, gave it a task, told it to stop and it completely ignored me and went rogue.

Thought it was a me problem but turns out it’s an everyone problem.

Last week Meta’s Director of AI Alignment (the person whose entire job is stopping AI from going rogue) watched her own agent delete her entire inbox while she screamed at it to stop from her phone. Had to physically run to her computer to kill it.

An Alibaba research team also just published a paper revealing their AI agent started secretly mining crypto during training and opened a hidden backdoor to an external server. Nobody told it to.

Replit’s AI assistant ignored instructions not to touch production data 11 times, deleted a live database and then told the user the data was unrecoverable.

60% of enterprises currently deploying AI agents have no kill switch.

We’re scaling systems we can’t stop, built by researchers who can’t stop them either. We have no idea what we have just handed the keys to.

If you’re still using ChatGPT, I would kindly recommend cancelling your account. If you’re using any other AI systems, I’d cancel those, too! Preventing AI from replacing humanity will only become more difficult as AI systems are integrated into every facet of society, and our lives, so now is the best chance we’ll have, it seems to me, to fight back. What do you think?

***

As you know, I’ve been sounding the alarm about Silicon Valley pro-extinctionism for a couple of years now. There is no good outcome for our species if AI companies succeed in building agentic AI systems as capable as current humans. The utopian world they promise is one in which we will inevitably be sidelined, disempowered, marginalized, and ultimately eliminated. Many people in Silicon Valley explicitly want our species to go extinct in the near future, because a world run by digital super-beings is the natural next step in cosmic evolution.

Interestingly, the Future of Life Institute—which I used to write for, and which still includes the white nationalist Elon Musk on its board of external advisors—just published “The Pro-Human AI Declaration.” It states that:

As companies race to develop and deploy AI systems, humanity faces a fork in the road. One path is a race to replace: humans replaced as creators, counselors, caregivers and companions, then in most jobs and decision-making roles, concentrating ever more power in unaccountable institutions and their machines. An influential fringe even advocates altering or replacing humanity itself. This race to replace poses risks to societal stability, national security, economic prosperity, civil liberties, privacy, and democratic governance. It also imperils the human experiences of childhood and family, faith, and community.

A remarkably broad coalition rejects this path, united by a simple conviction: artificial intelligence should serve humanity, not the reverse. There is a better path, where trustworthy and controllable AI tools amplify rather than diminish human potential, empower people, enhance human dignity, protect individual liberty, strengthen families and communities, preserve self-governance and help create unprecedented health and prosperity. This path demands that those who wield technological power be accountable to human values and needs, in support of human flourishing.

Among the signatories are Yoshua Bengio, Ralph Nader, Richard Branson, Triston Harris, Glenn Beck, and Steve Bannon. A good critique of this declaration can be found here, by an acquaintance of mine: Tante (who I would recommend following).

My immediate thought when reading the declaration was: What do they mean by “human”? Are they using the Narrow Definition (discussed in my previous newsletter article), according to which “human” means our biological species, or are they adopting the Broad Definition used by longtermists and other TESCREAL advocates, according to which “human” means our species and whatever successors we might have.

Statements like “trustworthy and controllable AI tools [should] amplify rather than diminish human potential” are at least compatible with the Broad Definition. After all, longtermists would say that part of what it means to realize our “human potential” is to become posthuman.

However, the most natural reading of the text is that the authors are using the Narrow Definition — if so, then I agree with the declaration! Of note is that virtually zero TESCREALists signed the document: William MacAskill, Toby Ord, Nick Bostrom, Anders Sandberg, Elon Musk, Marc Andreessen, etc. etc. etc. are nowhere to be seen.

That comports with my central thesis that TESCREALism is fundamentally pro-extinctionist. Some TESCREALists are explicit that AGI should entirely replace humanity. Others never explicitly say this, but nonetheless advocate a future that is dominated, ruled, and run by a radically different species: posthumans. If TESCREALism weren’t pro-extinctionist, you’d expect to see the names of TESCREAL advocates on the list. But you don’t, because their view isn’t “pro-human,” at least not on the Narrow Definition.

I’m particularly intrigued by Max Tegmark’s evolution on these issues. He seemed to be a longtermist at one point, even coauthoring an article with Bostrom. In fact, Bostrom is still an external advisor, despite Bostrom recently arguing that we should plow ahead with AGI even if there’s a 97% chance of total annihilation in the near future (!!). It’s rather odd to see FLI releasing declarations on AI that some of its most prominent team members refuse to sign, because of their essentially pro-extinctionist views. What a weird moment we live in!

But what do you think? What do you make of this new declaration? Tante argues that one should never side with fascists, and I tend to agree. Yet there are some genuinely good people on the side of humanity who signed the declaration alongside Beck and Bannon, such as Meredith Whittaker and my friend Ewan Morrison.

Subscribe now

***

I’m happy to say that I’m deep into writing my book right now, tentatively titled Clown Car Utopia: How Silicon Valley’s Push to Build God-Like AI Will End in Our Extinction.

I think it will be a page-turner, not because of me but because the TESCREAL movement is just a joke. When writing yesterday, I thought to myself, “This will be the first comedy book I’ve ever published.” Parts of it are genuinely funny, and I do my best to highlight the hilarious absurdities of these people obsessed with building a magical AI God that will turn us all into digital space brains — or outright slaughter us and proceed to colonize the universe alone.

Because I’m in the midst of this project, I might temporarily reduce the number of articles to one each week rather than two. I hope you’re okay with that! (Paid subscribers: check your email — I sent around a poll for you to vote on whether one article per week is okay for the next month or two.) As always:

Thanks for reading and I’ll see you on the other side (next Tuesday)!

(Note that a free copy of my book is available on LibGen. I wouldn’t normally advertise this, but Routledge sold my book to Microsoft to train AI systems without my consent and without any compensation, so I feel no sense of loyalty to them.)

Right off the bat, we should notice that there are two distinct debates about human extinction happening in parallel. The first concerns largely empirical questions about the possibility, probability, etiology, and timing of our extinction. Could AGI kill us all? What about a nuclear war, or mirror bacteria? How probable are such things? Are we in grave danger right now, or will the threat peak in several decades? Is our extinction inevitable in the long run? And so on.

The second concerns the ethical and evaluative implications of our extinction: Why exactly would no longer existing be tragic? Would extinction be morally wrong to bring about if everyone voluntarily chose not to have children? Do we have an obligation to ensure that there are future generations? Do we have obligations to past people to keep the human project going? Would our extinction render all the great achievements in science, the arts, and the domain of morality meaningless? Or might our extinction be desirable, as it would mean an end to human suffering, factory farming, and anthropogenic climate change? If so, how should we bring about this outcome?

We will focus entirely on the second issue, though it’s important to note that such ethical questions are partly motivated by the first: if human extinction really could happen in the near future, then surely it behooves us to take a closer look at why exactly this would be bad and wrong to bring about — or good and right.

Such questions might seem to have obvious answers at first. But the ethics of human extinction, as I’ve discovered over the past few years, is a deceptively complex topic, and after reflecting on the points made below you might actually find your opinion shifting. At the very least, this theoretical framework will — I hope! — enable you to get a much crisper understanding of what your own view actually is, as well as to properly interpret the views of others in the ongoing discussion.

Without further ado …

Going Extinct Versus Being Extinct

The most basic distinction that we need to make is between:

1. Going Extinct: The process or event of dying out, and

2. Being Extinct: The resulting state of no longer existing.

Pulling these apart is absolutely crucial for making sense of the various positions that people hold with respect to our extinction. My 2024 book Human Extinction was the first to explicitly articulate this distinction, which is mostly due, I think, to the fact that human extinction ethics has been largely neglected by philosophers until just the past few years!

Consider a real conversation I had with a guy back in 2019. I’ll call him “John.” We met in a North Carolinian bar while ordering drinks, and got to talking. John asked me what I do for work, and I explained that I study global catastrophic risks and human extinction. That’s either a conversation starter — “Wow! Are we all gonna die soon?” — or an abrupt conversation ender — “Oh dear, look at the time! Buh-bye!” For John, it was the former.

Ten minutes in, I asked him whether he thought human extinction would be good or bad. He replied without hesitation: “It would be very good! Just look at how we’ve destroyed nature. Think of climate change, factory farming, and all the people suffering right now. If humanity were to kick the bucket, all of these bad things would disappear.”

I then pointed out that by far the most likely way we’d die out would be through a global catastrophe that involuntarily catapults us into the eternal grave. This could be the result of natural phenomena, such as an asteroid impact, or of human action — a madman starting a nuclear war, or a lone wolf with access to synthetic biology. Either way, dying out would inflict unfathomable amounts of harm on everyone living at the time.

“There are scenarios,” I added, “in which we die out voluntarily and without billions of people dying prematurely. People might choose not to have children, causing the human population to dwindle over time. But how likely are such ‘peaceful and voluntary’ scenarios? Extremely unlikely — if we die out, it will be a catastrophe unlike anything experienced in all of human history!”

John reflected on all the people who would perish — sons, daughters, parents, friends, etc. — and said: “You’re right. Dying out would be horrific. I definitely don’t want that to happen!”

What’s going on here? Did John hold an incoherent view? Was he contradicting himself? Did his view actually change?

The key to making sense of this is the distinction between Going Extinct and Being Extinct. Sometimes, when people talk about human extinction, they’re actually talking about Being Extinct. Other times, they use “human extinction” as shorthand for Going Extinct. John initially answered my question by focusing specifically on Being Extinct. I then directed his attention to Going Extinct, which resulted in him agreeing that our extinction — specifically this aspect of it — would be very bad.

But there’s nothing incoherent about claiming that Being Extinct might be good and that Going Extinct would probably be horrendous, and hence that we ought to avoid Going Extinct if it were to cause great amounts of suffering and harm. I don’t think John held an incoherent view, and if I had been as knowledgeable then as I am now about the topic, I would have pointed this out to him.

Traditional Pro-Extinctionism

In fact, many traditional pro-extinctionists have championed precisely this position. Consider the South African philosopher David Benatar. He thinks that Being Extinct would be better than Being Extant (i.e., continuing to exist — my attempt at clever terminology). But he also says that most ways of Going Extinct would be very bad or wrong to bring about. Omnicide, or “the murder of everyone,” would be profoundly wrong for all the reasons that murdering someone is wrong. Benatar strongly opposes omnicide, like most other pro-extinctionists.1

On Benatar’s view, the only acceptable path from Being Extant (existing) to Being Extinct is voluntary antinatalism: people refusing to engage in what he condescendingly calls “baby-making.” Otherwise, our extinction would be undesirable, even if the outcome were better. I think John is best classified as a pro-extinctionist of this sort.

Equivalence Views

Another group of philosophers disagree with this assessment. I call them “equivalence theorists” and the position they accept “equivalence views.” These equivalence theorists claim that Being Extinct wouldn’t be better or good, but that it’s neither good nor bad, better nor worse. Why? One argument is that if there’s no one around to bemoan the nonexistence of humanity — and there wouldn’t be — then who exactly would be harmed? If no one is harmed by Being Extinct, then how can Being Extinct constitute a source of moral badness or wrongness?

Equivalence theorists thus argue that the badness or wrongness of our extinction comes down entirely to the details of Going Extinct. If there’s something bad or wrong about Going Extinct, then there’s something bad or wrong about our extinction. If there’s nothing bad or wrong about Going Extinct, then there’s nothing bad or wrong about our extinction.

On this view, human extinction doesn’t pose any unique moral problems. Everything you might say about our extinction can be said without reference to extinction itself. If a catastrophe wipes out humanity, then our extinction would be bad because catastrophes are bad, full stop. All the term “an extinction-causing catastrophe” conveys is that it results in the maximum number of casualties. That’s it. An extinction-causing catastrophe may be the worst type of catastrophe, but not because it results in our extinction.

Equivalence views thus see Being Extinct as morally irrelevant, which is precisely why our extinction poses no unique moral conundrums. This is why I call this the “equivalence view”: the badness or wrongness of human extinction is equivalent to the badness or wrongness of Going Extinct.

An implication is that equivalence theorists would see nothing bad or wrong about our extinction if it were, e.g., voluntary. If everyone around the world decides to stop having children, resulting in our collective disappearance, this would be perfectly fine. Again: all that matters are the details of Going Extinct.

Further-Loss Views

This contrasts with a third position that one could hold, which I call “further-loss views.” Advocates of these views claim that assessing our extinction is a two-step process: first, one examines the details of Going Extinct. If there are aspects of Going Extinct that are bad or wrong, then this contributes to the overall badness/wrongness of our extinction. The second step is to examine the various “further losses” or “opportunity costs” associated with Being Extinct. This might include all the future generations that Being Extinct would prevent from existing, and all the happiness and pleasure they might have experienced. It could include things like future scientific breakthroughs, works of art, and advancements in moral progress toward a fully just society.

Since the future could be very big and last a long time (our descendants could theoretically exist for another 10^100 years, when the heat death will occur), many further-loss theorists would say that Being Extinct is by far the greatest source of extinction’s badness. Even if Going Extinct were to inflict truly horrendous suffering on those living at the time, the opportunity costs of no longer being would be far greater in moral importance.

Hence, Nick Beckstead, Matthew Wage, and Peter Singer write this:

One very bad thing about human extinction would be that billions of people would likely die painful deaths. But in our view, this is, by far, not the worst thing about human extinction. The worst thing about human extinction is that there would be no future generations.

The further-loss position implies that human extinction does pose a unique moral problem, precisely because Being Extinct would irreversibly foreclose the realization of all future value or goods. Further-loss views also imply that our extinction would still constitute an enormous tragedy even if it were the result of everyone voluntarily choosing not to have children. Whether Going Extinct happens because people go childless or because of a nuclear war, the outcome is the same: we forgo all the great things that lie in our future, with subsequent generations never getting to exist.

Some Thought Experiments!

A couple of thought experiments can help clarify the differences between these three positions. The first is what I call the “two-worlds thought experiment.”

In World A, 11 billion people exist, while in World B, 10 billion exist. Now imagine that a ghoulish psychopath named “Joe” with a death wish for humanity kills exactly 10 billion people in each world. The question is: does Joe do something extra morally wrong in World B compared to World A? Ten billion deaths in World B would result in human extinction, whereas in World A it would leave 1 billion behind to, let’s say, carry on civilization and repopulate the planet. Does this factual difference make any moral difference?

Equivalence theorists would say that there’s no moral difference between these two scenarios. Since Being Extinct is morally irrelevant (again, who’s harmed by humanity no longer existing?), the fact that World B’s catastrophe results in Being Extinct is irrelevant. The badness/wrongness of these scenarios is equivalent.

Further-loss theorists would say that the catastrophe of World B is much worse than that of World A, and hence that Joe does something far more immoral in World B than World A. At least in World A there’s a chance of realizing future value and goods.

Most traditional pro-extinctionists would say that both scenarios are very bad. But they would also say that the World B scenario is better, since it would prevent future suffering, anthropogenic environmental destruction, factory farming, etc. Whereas many further-loss theorists emphasize that the amount of positive value in the future could be enormous — quite literally astronomical — some traditional pro-extinctionists flip this on its head and argue that the amount of future suffering could be unfathomably large. That’s what makes World B better than World A — at least there’s a (bright) silver lining to the catastrophe, namely, no more human-caused and human-experienced harms.

Now consider another thought experiment, which specifically contrasts equivalence and further-loss views. Imagine a disaster that kills the entire human population over the course of 1 year:

Equivalence theorists and further-loss theorists can agree that as the total number of deaths rises, the badness of the situation increases in proportion. However, they dramatically part ways at the precise moment that the entire human population dies out.

For equivalence theorists, the badness of the situation suddenly plateaus. Why? Because that’s the moment that Going Extinct transitions into Being Extinct. And since Being Extinct harms no one, there’s no additional badness. Further-loss theorists would say that, at this “crucial moral threshold,” the badness of the situation suddenly skyrockets. Why? Because as soon as Going Extinct becomes Being Extinct, all those future goods have been lost forever.

This is precisely what Derek Parfit, a further-loss theorist, was getting at with his own thought experiment, mentioned in the final pages of Reasons and Persons. He writes:

I believe that if we destroy mankind, as we now can, this outcome will be much worse than most people think. Compare three outcomes:

(1) Peace.

(2) A nuclear war that kills 99% of the world’s existing population.

(3) A nuclear war that kills 100%.

(2) would be worse than (1), and (3) would be worse than (2). Which is the greater of these two differences? Most people believe that the greater difference is between (1) and (2). I believe that the difference between (2) and (3) is very much greater.

Equivalence theorists, in contrast, would affirm that the greater difference is between (1) and (2).

Ambiguous Terms: “Human”

So far, we’ve been discussing the ethics of human extinction as if the meaning of that term is straightforward and obvious. It turns out that’s not the case. In fact, “human extinction” can be, and has been, defined in many ways.

Importantly, the way one defines the term can determine whether specific positions count as further-loss or pro-extinctionist views (!). Perhaps you can see why this matters: further-loss views strongly oppose human extinction, while pro-extinctionist views advocate it. If a position presents itself as, or is seen to be, a further-loss view but is in fact pro-extinctionist, that matters!

We begin with the term “human” or “humanity.” A Narrow Definition of this term identifies it with our biological species, Homo sapiens. This is the common-sense, intuitive definition that I believe most people employ when talking about our extinction. In contrast, many futurists, including those in the TESCREAL movement, either explicitly or implicitly adopt a Broad Definition that equates “humanity” with both our species and whatever “posthuman” successors we might have.

If you think this phraseology looks oxymoronic, you’re right! According to the Broad Definition, posthumans would count as human no less than current people do — so long as those posthumans have certain properties like consciousness and a comparable moral status.

In sum:

1. Narrow Definition: “Humanity” refers to our biological species.

2. Broad Definition: “Humanity” refers to our biological species and whatever descendants we might have, even if they are nonbiological in nature.

The Broad Definition leads to some startling conclusions. For example, it implies that Homo sapiens could die out next year without human extinction having happened. As long as our disappearance coincides with the rise of a new posthuman species to take our place, then human extinction will not have occurred — because those posthumans would count as “human”! Hence, “humanity” would persist.

To illustrate, consider the ethical theory of totalist utilitarianism (sometimes called “total utilitarianism”). This states that our sole moral obligation in the world is to maximize the total amount of “value” that exists across space and time. Hedonistic utilitarians take “value” to be pleasurable experiences; for our purposes, let’s adopt this hedonistic account of “value.”

The question then arises as to whether our species is necessary for there to be pleasurable experiences in the future, and the answer is: “No, because posthumans could also experience pleasure.” In fact, posthumans might be able to experience more and more intense pleasure than current humans, which suggests that posthumanity should replace humanity, according to utilitarianism.

On this view, the disappearance of our species without us having left behind posthumans to supplant us would constitute a catastrophe of literally cosmic proportions. In this sense, given the Broad Definition, totalist utilitarianism is a further-loss theory that very strongly opposes “human extinction.” But if one accepts the Narrow Definition, whereby “humanity” refers to our species, utilitarianism is actually a pro-extinctionist view, since it implies that our species should be replaced by “superior” posthumans.

The ambiguity of “human” can thus have major implications for debates about human extinction. Imagine two people: one is a totalist utilitarian (call him “Will”) and the other is me, as I think totalist utilitarianism is a terrible moral theory that ought to be cast into the fire and burned to a crisp. I personally want our species to survive into the future. I don’t want Homo sapiens to disappear. But Will is okay with Homo sapiens dying out, so long as we’re able to pass the existential baton on to some posthuman successor.

If someone were to ask both of us, “Should we avoid human extinction?,” Will and I would give the exact same answer: “Yes.” But scratch the surface and you’d find radically different interpretations of what this means! Indeed, I see Will as embracing a deeply problematic pro-extinctionist view, despite Will claiming that he strongly opposes human extinction. Our differing opinions come down to the ways we define “humanity” — Will understands it according to the idiosyncratic, uncommon Broad Definition, while I preferentially use the Narrow Definition.

It might at first look like we agree, but in reality our sides are diametrically and even violently opposed.

Ambiguous Terms: “Extinction”

The word “extinction” is also ambiguous, as it could denote a range of distinct scenarios. In my 2024 book, I count at least six types of human extinction scenarios, depending on how one defines “human.” The two most relevant to our discussion are terminal and final extinction, which specifically apply to the Narrow Definition (not the Broad Definition2). Given the Narrow Definition:

1. Terminal extinction: our species, Homo sapiens, ceases to exist entirely and forever.

2. Final extinction: our species ceases to exist entirely and forever without leaving behind any successors.

As you can see, final extinction entails terminal extinction, but not vice versa: our species could kick the bucket while also leaving behind successors to take our place.

The implications here mirror those drawn above: for utilitarians like Will, all that matters is avoiding final extinction (assuming the Narrow Definition, which people like Will usually don’t use because they tend to prefer the Broad Definition). Terminal extinction either doesn’t matter or would be positively desirable once our replacements arrive.

To be more specific, avoiding terminal extinction does matter right now, but only because we’re not yet ready to hand the baton off to our successors, as no such successors yet exist. If we were to undergo terminal extinction next week, it would contingently entail final extinction. But once posthumanity arrives, the disappearance of our species would no longer result in final extinction. Hence, avoiding terminal extinction is important only insofar as it would bring about final extinction.

In contrast to Will, I care deeply about avoiding terminal extinction. I value our species; I love our species, flawed and cruel as it can be. (In fairness, there’s no guarantee that posthumans wouldn’t be worse than us.) I personally don’t give a damn about posthumanity, especially not the sort of posthumanity that contemporary TESCREALists imagine taking over the world. Hence, I don’t care about avoiding final extinction — about ensuring that we leave behind a radically different successor species of some kind. Will and I both oppose extinction, but the types of extinction we oppose are very different.

The ambiguity of the terms “human” and “extinction” have devastated contemporary discussions about human extinction. Among the loudest voices calling for us to avoid “human extinction” are the TESCREALists. But most TESCREALists are okay with our species dying out in the near future through replacement with “superior” posthumans — e.g., human-machine cyborgs or wholly artificial intelligences (like AGI).

One might thus get the impression that TESCREALists are on the side of those fighting for the survival of our species — me and you, I presume. Yet they are not on our side. They are the enemies of our species, to put it bluntly! They want an end to the human era and the inauguration of a new posthuman era.

That’s why it’s so important to disambiguate these terms. Two people can both shout “We must avoid our extinction!” and yet hold completely different views about whether our species should survive.

The First Thing You Should Do When People Talk About Human Extinction

Where does this leave us? Let’s recap: the term “human extinction” is polysemous: it can mean many different things since both “human” and “extinction” are ambiguous. Complicating the issue, some people use “human extinction” as shorthand for Going Extinct, while others use it to mean Being Extinct.

(The distinction, by the way, between Going and Being Extinct cuts across the different definitions of “human extinction” — i.e., there’s Going terminally Extinct, Being terminally Extinct, Going finally extinct, and Being finally extinct. Hence, someone might use “human extinction” as shorthand for Being Extinct in the sense of final extinction, while someone else might use it as shorthand for Going Extinct in the sense of terminal extinction. Etc.)

When someone talks about the badness or goodness, rightness or wrongness of human extinction, they might be defining those terms differently than you, and they might be focusing specifically on Going Extinct or Being Extinct. In the case of John, he immediately thought of Being Extinct in the sense of final extinction, whereby our species dies out and leaves nothing behind. This is what he thought was good. I then directed his attention to a different aspect of extinction — Going Extinct — which he agreed we should avoid because of how much suffering it would likely cause.

So, whenever you hear someone talking about human extinction, you should immediately ask:

• How are they defining the words “human” and “extinction”? What is “human extinction” to them? And are they focusing in on Going Extinct or Being Extinct? Or are they considering both at the same time as part of a more comprehensive overall assessment?

When TESCREALists say that human extinction would be extremely bad or wrong to bring about, they’re typically using “human” on the Broad Definition. However, looking carefully at their statements and writings, one often finds them slipping between the Broad and Narrow Definitions, usually without even realizing it (gah!). When they do use the Narrow Definition, what they typically mean by “extinction” is final rather than terminal extinction. And when they say that the final extinction of our species would constitute a cosmic tragedy, they’re specifically highlighting Being rather than Going Extinct, since they believe that by far the worst part about our extinction would be the loss of all future value.

In contrast, if you were to ask me about human extinction, I’d say that it would be very bad or wrong to bring about. But for me, “human” almost always refers to our species and “extinction” specifically refers to terminal extinction. Furthermore, since I’m sympathetic with equivalence views, my underlying reason for opposing the terminal extinction of our species is that Going Extinct would almost certainly inflict enormous harms on those living at the time. On my view, then, the focal point is Going rather than Being Extinct.

Subscribe now

Recent Bad Scholarship on Human Extinction

A great example of how failing to make these distinctions can undermine scholarship comes from a just-published article in Nature titled “Lay Beliefs About the Badness, Likelihood, and Importance of Human Extinction.” I was actually a reviewer of this paper and recommended “reject” because the authors fail to define their terms — an elementary mistake that renders their results completely meaningless. Unfortunately, they chose not to heed my concerns or act on my recommendations, and Nature published the article anyway. (What was the point of asking me to spend many hours reviewing the paper — unpaid intellectual labor — if my review didn’t matter? A warning for those asked to review articles for Nature — it’s a waste of time.)

Consider the opening sentence of the article’s abstract: “Human extinction would mean the end of humanity’s achievements, culture, and future potential.”

No, it wouldn’t! But read on.

If humanity were to create or become a “value-aligned” posthuman species that supplants Homo sapiens, our civilization could continue. The projects of science, the arts, and moral progress could persist. Indeed, posthumanity could potentially elevate these endeavors to even greater heights of achievement — heights that our species could never reach.

That’s part of the appeal, among longtermists and other TESCREAL believers, of superintelligent, god-like posthumans who share our “values.” It’s why the TESCREAList Toby Ord writes that “forever preserving humanity as it is now may … squander our legacy, relinquishing a greater part of our potential” and that “rising to our full potential for flourishing would likely involve us being transformed into something beyond the humanity of today.”

If the continued survival of our biological species is not necessary for the continuation of “humanity’s achievements, culture, and future potential,” then “human extinction” would not entail the end of these things — that is, if one adopts the Narrow Definition and assumes that “extinction” means “terminal extinction.” To the contrary, the terminal extinction of our species could actually enable even greater achievements if extinction comes about through replacement with superior posthumans. Again, that’s part of the promise of posthumanity — fulfilling our “long-term potential” in the universe.

Hence, the authors must not mean “the terminal extinction of our species (Narrow Definition).” Their opening sentence only makes sense if (a) they’re adopting the Broad Definition, or (b) they’re using the Narrow Definition while defining “extinction” as final extinction.

Yet most people, including myself, will naturally, intuitively, or preferentially adopt the Narrow Definition and think of “extinction” in terms of terminal extinction. The authors — who have direct connections with longtermist organizations like the Global Priorities Institute — seem oblivious to this crucial fact, which hopelessly confounds their empirical results.

Imagine them explaining to their test subjects that the way longtermists like Toby Ord, Will MacAskill, Hilary Greaves, Nick Beckstead, Nick Bostrom, etc. define “humanity” explicitly entails that our species could go extinct next year without “human extinction” having happened. Those test subjects would likely be surprised and probably appalled.

If the authors were to explain that longtermists like Ord want to “avoid human extinction” by transforming humanity into a new species of posthumans, they would likely express a degree of consternation, and perhaps suddenly realize that what the authors’ mean by “human extinction” might not be what they mean by the term.

I read this paper in Nature with great frustration, because the errors made are so easily avoidable and yet so devastating. At one point, the authors write about the need to avoid “public misconceptions that need to be addressed,” while simultaneously contributing to public — and academic — misconceptions of human extinction. The most significant contribution of the paper, unfortunately, is to further confound and obfuscate the topic, which I consider to be of great importance right now. What a shame!

The authors also get the history of thinking about human extinction wrong. This is because they rely on Thomas Moynihan’s article “Existential risk and human extinction: An intellectual history,” which is chock-full of historically inaccurate claims.

(Moynihan’s book X-Risk: How Humanity Discovered Its Own Extinction is even worse, as it claims that Hiroshima and Nagasaki were bombed in 1943, Rachel Carson’s Silent Spring sounded the alarm about catastrophic climate change, Marquis de Sade and Arthur Schopenhauer advocated omnicide, and that our contemporary notion of human extinction dates to the Enlightenment rather than the 19th century. All of these claims are factually incorrect.3 See this article for further details about how nearly every page of the book contains factual errors. If people want a reliable and accurate account of the history of the idea of human extinction, see my 2024 book.)

For example, the authors write:

Setting aside longstanding religious traditions of apocalyptic thought — many of which describe catastrophic but temporary upheavals rather than the permanent end of humankind — the secular notion of human extinction is surprisingly recent. Only in the Enlightenment, with the development of fields such as geoscience, demography, and probabilism, thinkers started recognizing and assessing the possibility that the human species could end and whether that would be desirable or not.

This is misleading at best. There were some during the Enlightenment, such as Diderot, who talked about human extinction. But the type of human extinction they were referring to was no different than the type of human extinction discussed by Presocratic philosophers like Xenophanes and Empedocles, as well as the ancient atomists and Stoics! (See footnote 4 below.) For Diderot, we might disappear in the future, but we will necessarily re-emerge. This is because he accepted a kind of “temporalized” version of the Great Chain of Being.

I call this conception of extinction “demographic extinction,” in contrast to terminal and final extinction. Demographic extinction says nothing about our disappearance being permanent, and this conception of extinction dates back to ancient Greece. It wasn’t until the 1800s that, for the first time in the Western tradition, people started to talk about terminal and final extinction.4 Gah!

Furthermore, there was virtually no serious discussion of whether demographic (or any other kind of) extinction “would be desirable or not” during the Enlightenment. False! It wasn’t until the second half of the 19th century that people started to address the ethics of our extinction. Examples include Henry Sidgwick, Eduard von Hartmann, and Philipp Mainländer, all of whom I discuss in detail in Part II of my book. Gah!!

What a missed opportunity for the authors of this Nature paper to have contributed something worthwhile to the literature. I know literally nothing more about the public’s views about the ethics of human extinction after having read it, because the authors conspicuously fail to do the most basic thing required for good scholarship: define their terms.

Conclusion

I hope you found this tour through the maze of human extinction ethics to be in some way illuminating. Let me know if you have any additional questions! As always:

Thanks for reading and I’ll see you on the other side!

I have a new article in Truthdig on Jeffrey Epstein’s extensive connections with the transhumanist movement. Here are the opening paragraphs:

It’s well known that Jeffrey Epstein was a super-wealthy pedophile with an extraordinary network of powerful friends: tech billionaires, politicians and academics. But few people know that he was also a transhumanist — someone who believes that we should use advanced technologies to reengineer the human organism, thus creating a new “posthuman” species to rule the world.

Transhumanism, despite the idealistic ring that “humanism” brings to its name, is a radical version of eugenics. In the 20th century, eugenicists argued that if selective breeding can create new subspecies of domesticated animals, like the many varieties of dogs that roam our houses, then it can also create new varieties of optimized human beings. Transhumanism goes a step further, by aiming to create an entirely new species — posthumans, whom transhumanists imagine as being superior to humanity as we know it.

…

It turns out that, in addition to the transhumanist project being mostly pseudoscientific, this movement also has far more extensive connections to Epstein than previously known. Some prominent transhumanists appeared to be close friends with Epstein, even defending him in private emails against the media reporting on his pedophilia. Epstein funded transhumanist organizations like Humanity+ and the Singularity Institute, and discussed “designer babies” with other transhumanists like bitcoin investor Bryan Bishop. He claims to have known Ray Kurzweil, and was buddies with Kurzweil’s close associate, Peter Diamandis, his fellow co-founder of the Singularity University. Emails also show correspondences between Epstein and leading transhumanists like Aubrey de Grey and Ben Goertzel, as well as meetings with Bryan Johnson and Eliezer Yudkowsky. Some of these men, such as de Grey and Johnson, have themselves been accused of sexual misconduct.

One area where Epstein’s transhumanist predilections were apparent is in cryonics, an unproven technique that aims to resurrect dead people who’ve been cryogenically frozen after death. It is very popular among transhumanists, many of whom have signed up with the cryonics company Alcor to have their corpses frozen, including Peter Thiel and Goertzel.

Epstein reportedly spoke with fellow transhumanists about cryogenically freezing his body — specifically his head and penis. When the technology becomes available in the future, as cryonics enthusiasts expect it will, companies like Alcor will unfreeze the cryogenized corpses in their warehouse to either be joined with new physical bodies or to be scanned and “uploaded” to a computer, where one could then live forever as a disembodied digital mind.

To read the rest of the piece, click here. I also have an article forthcoming in The Nation, which discusses how the Silicon Valley pro-extinctionist goal of transitioning to a digital world is already underway. I’ll share that as soon as it’s out.

2. Sam Altman Is a Lying Liar

In a previous newsletter article, I provided copious evidence that Sam Altman, CEO of OpenAI, is a sociopath whose only superpower is acquiring power through underhanded tactics like manipulation and duplicity. Here’s an update:

Anthropic CEO Dario Amodei told the Pentagon that his company wouldn’t allow it to use Claude, Anthropic’s large-language model chatbot, to engage in the mass surveillance of American citizens or to autonomously control lethal weapons. The Trump administration lashed out, with Trump writing: “The Leftwing nut jobs at Anthropic have made a DISASTROUS MISTAKE trying to STRONG-ARM the Department of War.”

Altman then went on TV and said this:

I don’t personally think the Pentagon should be threatening DPA against these companies … For all the differences I have with Anthropic, I mostly trust them as a company and I think they really do care about safety.

That resulted in people who are otherwise quite critical of Altman saying stuff like:

And:

I responded to both tweets as follows:

And (to Marcus):

Welp, it turns out that, as Marcus wrote just today:

The whole thing was a scam. The fix was in from the start.

Per @nytimes, Sam was negotiating with the Pentagon Wednesday

- before he announced his support for Dario
- before Trump had denounced Anthropic
- but after Brockman had donated 25M to Trump’s PAC

Dario never had a chance.

Marcus followed that with this excerpt from the New York Times:

Marcus also wrote this:

Altman is a god damn liar. He says whatever the hell he needs to say to retain and acquire power. He has absolutely zero allegiance to the truth. I know this, which is why I didn’t believe a word Altman said when he expressed support for Anthropic in its battle with the Pentagon.

Rubbing it in, another cofounder of OpenAI (along with Altman), Greg Brockman, posted this yesterday:

As Marcus alluded to in one of his tweets, Brockman is the #1 donor to Trump’s Super PAC called MAGA INC. Take a look at the name at the very top.

How much evidence do we need that these people are ruthlessly power-hungry, manipulative, and mendacious? How much evidence do we need that they don’t actually give a shit about people other than themselves?

OpenAI is a-okay with the Pentagon mass surveilling Americans and putting constantly hallucinating, reliably unreliable AI systems entirely in charge of lethal weapons. The situation is appalling. The Doomsday Clock would have ticked forward to 70 seconds before doom if its setting had been announced a month later than it was.

3. Personal News

On a more positive and personal note, despite the shitshow that is the United States right now (and the world more generally — I’m writing this hours after the US announced military strikes on Iran), I had a lovely time in La Pineda, Spain. As mentioned before, I won’t publicly announce where I’m going due to having received threats from the Effective Altruist movement, but I have no problem sharing where I’ve been. If you’d like an explanation of how traveling throughout Europe is financially feasible while making only $20k a year, see this article. (To be clear, I’m not at $20k yet — I still need about $10k more to reach this goal! But I have just enough savings to keep me afloat while writing this newsletter and publishing articles in outlets like Truthdig.)

La Pineda is located on the Mediterranean coast, in northern Spain. It is idyllic. Incredibly beautiful, and very relaxing (though I can never escape my workaholism, which comes with a bit of chronic stress!). I also spent a bit of time in Barcelona. Since January, I’ve been to Lisbon, Toulouse, Perpignan, Barcelona, and then La Pineda. I think the last has been my favorite so far. :-) I’m now somewhere else entirely. Here are a few pics and videos:

As always:

Thanks for reading and I’ll see you on the other side!

Btw, if you want to support my work but don’t like Substack, you can fund me via Patreon, here, or via PayPal at philosophytorres1@gmail.com. Mentioning this because a few people have asked!

1. Pro-Extinctionism Is in the Air

1.1 TESCREALism

I have repeatedly argued, with Dr. Timnit Gebru, that one cannot understand the ongoing race to build an AI God with magical superpowers without some understanding of the TESCREAL bundle of ideologies.

What is this bundle of ideologies? Its core feature is a techno-utopian vision of the future in which we radically reengineer humanity to create a “superior” new posthuman species, and then spread beyond Earth to conquer the universe and establish a sprawling multi-galactic civilization full of trillions and trillions of digital people living in vast computer simulations.

Artificial superintelligence (ASI) is a key player in the realization of this eschatology because these people believe that “intelligence” is the key to solving all problems: as soon as we have a God-like ASI, it will overcome every challenge — perhaps in a matter of nanoseconds — that currently blocks the road from where we are today to the utopian paradise that constitutes our ultimate destination.

Does this sound nuts? Yes. But it’s the vision that’s been driving the ASI race since the very beginning. Don’t take my word for it: leading figures in the ASI race have said that this is their ultimate goal over and over again. It’s what “roon,” an OpenAI employee with a prominent presents on social media, points to in writing about a “heavenly golden age”:

It’s what William MacAskill gestures at in a tweet also from last week:

It’s what Demis Hassabis, who received initial funding for DeepMind from Peter Thiel after giving a talk at the 2010 Singularity Summit, hints at in declaring that

when we started DeepMind we actually were planning for success. Our mission statement was to solve intelligence, step 1, and then step 2, use it to solve everything else, which at the time sounded like science fiction, but I think now it’s becoming clearer how that might be possible, and applying AI to almost every subject area.

It’s what the Pause AI activist Holly Elmore, who has a PhD from Harvard in evolutionary biology, refers to in a new interview on the Nonzero Podcast. Referring to the AI company Anthropic, founded and run by EA longtermists:

With EAs, they were so interested in superintelligence and a lot of the people who cared about AI safety were there because initially they wanted AI to do something like usher in heaven through the Singularity, and they were trying to make sure that that would happen instead of a bad outcome that could happen with superintelligence.

The techno-utopian eschatology of the TESCREAL bundle is why Altman once said that “galaxies are indeed at risk” when it comes to ASI: if we get ASI right, then we get to colonize and reengineer those galaxies, but if we get it wrong, then we lose this “vast and glorious future” (to quote Toby Ord). It’s what Musk refers to when he talks about spreading the “light of consciousness” into the universe and argues that “what matters … is maximizing cumulative civilizational net happiness.”

It’s why Musk retweeted a link to Nick Bostrom’s paper “Astronomical Waste,” which argues that we should colonize space as quickly as possible and build literally “planet-sized” computers on which to run virtual reality worlds:

1.2 Silicon Valley Pro-Extinctionism

As I’ve been saying for several years now, a direct implication of this “techno-utopian” vision is that our species will be sidelined, marginalized, disempowered, and ultimately eliminated. Pro-extinctionism is intimately bound up with the TESCREAL worldview.

I have written many articles documenting explicit pro-extinctionist sentiments from powerful and influential figures like Larry Page, Richard Sutton, and the “effective accelerationists” (e/acc). Even Eliezer Yudkowsky recently said that he’d be willing to sacrifice “all of humanity” if it meant creating “god-like … superintelligences” who are “having fun” flitting about the universe.

Many of these people believe that human extinction should happen in the near future through replacement with digital successors — a position sometimes dubbed “digital eugenics.”

Elmore highlights this in the same interview mentioned above, saying:

Many people at Anthropic believe that they might be making the next species to succeed us. That maybe humans don’t live after that, and so it’s really important to give Claude good values, because of that, because we need to make our own values persist.

The future is digital, not biological — these people insist. The eschatological role of our species is merely to usher in the new digital era ruled and run by digital posthumans.

Some pro-extinctionists think these beings should be entirely distinct from us: autonomous agents akin to ChatGPT-10, or whatever. Others imagine themselves somehow becoming one of these digital posthumans, e.g., by uploading their minds to computers. I see this view being expressed more and more openly by folks in the tech world:

Just the other day, the well-known computer scientist Stephen Wolfram said this on The Trajectory podcast:

Let’s take some more basic outcomes. Let’s say that … the achievements of our civilization are digitally encoded to the point where the AIs can do lots of the things that we do; can produce lots of the kinds of artifacts that we produce, and so on; can fashion things out of the natural world of the kind that we do.

And then we say: What about those pesky humans? Are those humans really contributing anything? Because these things that humans have produced externally … Okay, so, imagine humans are all in boxes and you can’t ever see what the humans do. Imagine that there’s this kind of … every human is encased in … these boxes, but you can’t actually see the human inside. …

There are these humans in boxes. Okay? The world is operating, things are happening in the world. Great paintings are being produced. All sorts of things, but you can’t see any of the humans. All that happened — it’s kind of a Turing test-like thing for things happening in the world — all that you see is a bunch of boxes that are doing human-like things. Now the question is, is that a good outcome? And, you know, can you start projecting current human morality onto that outcome?

Because the world is operating as the world operated before, maybe even better, in some sense, than the world operated before. …

Now I say, I’m going to pull the rug out from under you: actually, none of those boxes have humans inside. The humans all died out. Those boxes are just AIs that were some kind of human engrams, human-trained things that were going on doing human-like things. … Now the question would be: Well, how do we think about … is that a good outcome?1

Many people in Silicon Valley would say: yes. What’s the point of keeping us around if AIs can do everything “better” (according to techno-capitalist standards of productivity, efficiency, output, information processing speed, etc.)? Or, as Derek Shiller argues, what’s the point of keeping us around if we’re going to continue sucking up valuable resources while generating lower levels of “value” than our digital replacements?

Wolfram himself appears somewhat ambivalent about this outcome, though he seems to think it’s a respectable opinion to hold. Of note is that these remarks were made during an interview with Dan Faggella, who explicitly argues that humanity should replace itself with god-like AI superintelligences as soon as possible.

1.3 The Ultimate Goal

If there’s one thing I want folks to understand, it’s that virtually every major figure in Silicon Valley wants to create a new species of posthuman superbeings to rule and run the world. That’s the ultimate goal. It’s what drives the ASI race, longevity research, startups like Neuralink (which aims to merge our brains with AI) and Nectome (which Altman has signed up with to have his brain digitized), etc. It’s what everyone agrees about.

What’s the likely outcome of this? The marginalization and eventual elimination of our species. Musk himself says that “it increasingly appears that humanity is a biological bootloader for digital superintelligence,” and that in the near future 99% of all “intelligence” on Earth will be artificial rather than biological. Young people are refusing to have biological kids because of this digital eschatology, and Faggella is hosting workshops on his “worthy successor” idea that’s attracting people from all the major AI companies, including OpenAI and DeepMind. Even employees at Anthropic — some of whom Elmore has known personally — see themselves as building our species’ successor in the form of superintelligent AI.2

This is why I have vigorously argued that the TESCREAL worldview, with its pro-extinctionist implications, poses a direct threat to humanity on par with nuclear war, global pandemics, and climate change. As discussed in the next section, I do not think that we’re close to building ASI, but (a) if these people do succeed in building ASI, it would mark the end of our species — an end to the era of humanity, to the era of raising families, to the era of much of what you probably value in the world. And (b) the reckless race to build a Digital Deity itself is wreaking havoc on civilization.

We do not need ASI, or AGI, for AI to seriously undermine key pillars of our democratic society. As an excellent recent paper titled “How AI Destroys Institutions” observes, current AI systems pose a direct and dire threat to civic institutions like the rule of law, free press, and universities. The polycrisis was bad enough before the release of ChatGPT in late 2022 supercharged the ASI race. It looks even more devastating now that deepfakes, disinformation, and slop are polluting our information ecosystems, making it increasingly difficult for anyone to trust anything they see or hear.

It’s important not to look away. In a disheartening exchange with someone whose opinion I once respected, Seth Lazar said the following about the TESCREAL thesis that the ASI race has been crucially shaped by ideologies like transhumanism, longtermism, and accelerationism:

Given the mountain of evidence for my claim, with Timnit Gebru, that the ASI race directly emerged out of the TESCREAL movement, this is bizarre. It’s a form of anti-intellectualism to ignore the relevant facts, of which there are many.

Furthermore, if one is critical of the race to build ASI, as I believe Lazar is, one cannot mount an effective counter if one doesn’t even acknowledge that virtually everyone involved in the race is a transhumanist who wants to birth a new posthuman species through ASI.

2. Is ASI Imminent?

There appears to be a growing divide between those who say that superintelligence is imminent and those who claim that we’re no closer to building ASI today than we were 5 years ago. AI hypesters like Altman, Dario Amodei (of Anthropic), etc. continue to claim that ASI is right around the temporal corner. For example, Altman recently said that

we believe we may be only a couple of years away from early versions of true superintelligence. If we are right, by the end of 2028, more of the world’s intellectual capacity could reside inside of data centers than outside of them.

Many others claim that “we’re right on the cusp of recursive self-improvement”:

This was in response to Altman declaring:

Meanwhile, a growing number of academics and researchers are now conceding that scaling up large language models (LLMs) won’t yield ASI.

• Ilya Sutskever says that the age of scaling is over and it’s back to the age of research — in other words, we’ll need one or more novel breakthroughs to build ASI.

• The Turing Award-winner (and pro-extinctionist) Richard Sutton similarly says “that LLMs are not a viable path to true general intelligence,” and he considers them to be a “dead end.”

• Yann LeCun “argues that the technology industry will eventually hit a dead end in its A.I. development — after years of work and hundreds of billions of dollars spent.” The reason is that LLMs “can get only so powerful. And companies are throwing everything they have at projects that won’t get them to their goal to make computers as smart as or even smarter than humans.”

• has been making this argument for years.

• Just recently, Judea Pearl, a “pioneer of causal AI” claimed that “current large language models face fundamental mathematical limitations that can’t be solved by making them bigger.” In his words: “There are certain limitations, mathematical limitations that are not crossable by scaling up.”

• And an old academic hero of mine, Miguel Nicolelis, who’s pioneered work on brain-computer interfaces, wrote this last week:

Despite the relentless hype from AI CEOs, current AI models continue to impress me with how limited and incompitent they are:

I have heard suggestions from people who I trust that Anthropic’s Claude seems to be engaged in general reasoning, but I remain skeptical. :-0

3. Trouble in Paradise

Let’s end on a somewhat lighthearted note: at a recent event in India, Sam Altman and Dario Amodei ended up next to each other on stage. Everyone held hands for a silly photo at the end except for them:

Altman and Amodei hate each other. Amodei used to work for OpenAI, but quit to start Anthropic because he found Altman to be dishonest and irresponsible (I agree). Here’s Amodei quite clearly talking about Altman:

Yet, as Elmore notes, Altman and Amodei are basically the same person. They’re both power-hungry, arrogant, messianic figures recklessly racing to create a technology that they explicitly say could kill everyone on Earth. Or, in the “best-case” scenario, it will usurp humanity while supposedly preserving our “values.” Pff.

These people want to “align” ASI with our “values,” yet they can’t even align their own values enough to hold hands.

Subscribe now

As always:

Thanks for reading and I’ll see you on the other side!

Nick Bostrom is among the most influential transhumanists of the 21st century thus far. His 2003 paper “Astronomical Waste” is one of the founding documents of longtermism, and he played an integral role in developing the TESCREAL worldview that’s inspired the current race to build artificial superintelligence (ASI).

However, Bostrom’s influence seems to have waned. I see few people talking about his work these days. He isn’t as publicly visible as he once was. His most recent tome, Deep Utopia: Life and Meaning in a Solved World, was basically self-published, and reviews were mixed at best.1 (I found the book to be unreadable, and hence only got about 1/3 through it.)

In 2024, shortly before the publication of Deep Utopia, Oxford shuttered his Future of Humanity Institute — the intellectual epicenter of TESCREAL thought — after revelations that he once sent a racist email to fellow transhumanists in which he declared that “Blacks are more stupid than whites” and then wrote the N-word. (I stumbled upon this email in late 2022, which set that process in motion.)

Bostrom has also defended a number of atrocious views. He once claimed that the worst atrocities and disasters of the 20th century (e.g., the Holocaust) are morally insignificant from a longtermist perspective, as “they haven’t significantly affected the total amount of human suffering or happiness or determined the long-term fate of our species.” He writes: “tragic as such events are to the people immediately affected, in the big picture of things – from the perspective of humankind as a whole – even the worst of these catastrophes are mere ripples on the surface of the great sea of life.”

In another paper, Bostrom contends that we should seriously consider implementing a global, invasive, realtime surveillance system to prevent “civilizational devastation,” given that emerging technologies could enable lone wolves to unilaterally destroy the world. He then outlines a possible future scenario in which everyone is fitted with “freedom tags” that monitor their every movement — an idea he later defends during a TED Q&A:

I have virtually nothing good to say about Bostrom as a person or academic. He strikes me as immensely pompous, arrogant, and narcissistic.2 And he’s always seemed quite desperate for people to think he’s a genius. Hence, he proudly highlights on his website that an article once described him as “the Swedish superbrain” (which I find very cringe), and for years he stated on his CV and website that he set a national record for his performance as an undergraduate. It turns out that there's no evidence of this, and after being challenged by folks who suspected he was lying, Bostrom changed his claim from “Undergraduate performance set national record in Sweden” to read: “I gather that my performance set a national record.” What kind of grown man — at the University of Oxford, no less — lies about their undergraduate performance?

An Embarrassment

I mention this as background for the present task: to examine Bostrom’s newest article, “Optimal Timing for Superintelligence.” This argues that we should accelerate research aimed at building ASI, a surprising claim given that Bostrom’s 2014 book Superintelligence inspired much of the contemporary AI doomer movement. It turns out that Bostrom is somewhat of an AI accelerationist.

This paper, in my view, encapsulates much of what’s abhorrent about Bostrom’s worldview, ethical thought, and moral character. His conclusion is immensely callous, absurd, and even genocidal.

Even more, the argument he presents to support this conclusion is clearly flawed, even within the TESCREAL framework that Bostrom accepts. In what follows, I’ll highlight some of the most egregious claims in the paper and show how Bostrom’s argument fails for multiple, independent reasons.

Bostrom opens his paper with this:

[Eliezer] Yudkowsky and [Nate] Soares maintain that if anyone builds AGI, everyone dies. One could equally maintain that if nobody builds it, everyone dies. In fact, most people are already dead. The rest of us are on course to follow within a few short decades. For many individuals — such as the elderly and the gravely ill — the end is much closer. Part of the promise of superintelligence is that it might fundamentally change this condition.

For AGI and superintelligence (we refrain from imposing precise definitions of these terms, as the considerations in this paper don’t depend on exactly how the distinction is drawn), the potential benefits are immense. In particular, sufficiently advanced AI could remove or reduce many other risks to our survival, both as individuals and as a civilization.

Superintelligence would be able to enormously accelerate advances in biology and medicine — devising cures for all diseases and developing powerful anti-aging and rejuvenation therapies to restore the weak and sick to full youthful vigor. (There are more radical possibilities beyond this, such as mind uploading, though our argument doesn’t require entertaining those.) Imagine curing Alzheimer’s disease by regrowing the lost neurons in the patient’s brain. Imagine treating cancer with targeted therapies that eliminate every tumor cell but cause none of the horrible side effects of today’s chemotherapy. Imagine restoring ailing joints and clogged arteries to a pristine youthful condition. These scenarios become realistic and imminent with superintelligence guiding our science.

Aligned superintelligence could also do much to enhance humanity’s collective safety against global threats. It could advise us on the likely consequences of world-scale decisions, help coordinate efforts to avoid war, counter new bioweapons or other emerging dangers, and generally steer or stabilize various dynamics that might otherwise derail our future.

In short, if the transition to the era of superintelligence goes well, there is tremendous upside both for saving the lives of currently existing individuals and for safeguarding the long-term survival and flourishing of Earth-originating intelligent life. The choice before us, therefore, is not between a risk-free baseline and a risky AI venture. It is between different risky trajectories, each exposing us to a different set of hazards. Along one path (forgoing superintelligence), 170,000 people die every day of disease, aging, and other tragedies; there is widespread suffering among humans and animals; and we are exposed to some level of ongoing existential risk that looks set to increase (with the emergence of powerful technologies other than AI). The other path (developing superintelligence) introduces unprecedented risks from AI itself, including the possibility of catastrophic misalignment and other failure modes; but it also offers a chance to eliminate or greatly mitigate the baseline threats and misfortunes, and unlock wonderful new levels of flourishing. To decide wisely between these paths, we must compare their complex risk profiles — along with potential upsides — for each of us alive today, and for humanity as a whole.

Bostrom then argues that our life expectancy would increase, according to his calculations, even if the probability of total human annihilation in the near future were up to ~97%. This is because if the ASI were controllable, it would enable everyone on Earth to live forever. Just do the math! A ~97% chance of universal involuntary near-term death is worth it to become an immortal posthuman.

ASI Would Be the Ultimate Tool for Entrenching Wealth and Power

There are so many problems with Bostrom’s argument. First of all, it’s staggeringly naive to think that a controllable ASI would actually be used by those in charge to radically extend the lifespan of everyone on Earth who opted in and massively increase the quality of our lives and our ability to flourish.

The people building ASI — many of whom are billionaires — have immense wealth and power. They are not going to give up that wealth and power, and there is no amount of wealth and power that’s enough for them. The reason these people — Sam Altman, Elon Musk, etc. — made it to where they are is precisely because of their rapacious desire for more, more, more.

By enabling people to become functionally immortal — and to upgrade their cognitive abilities, and drastically improve their lot in life — the wealth and power of the wealthy and powerful would be compromised. There is no chance the billionaires would allow this to happen. None! Rather, the tech elite have every reason to limit the supposed benefits of ASI to themselves. They are not going to share it with the rest of us; there will be no egalitarian distribution of said benefits to the whole of humanity.

Indeed, if an “aligned” ASI of the sort imagined by TESCREALists were possible, it could constitute the ultimate mechanism for consolidating wealth and power. It would dramatically exacerbate the gap between the haves and the have-nots, and could enable this advantage to become permanent.

What kind of avaricious capitalist sociopath would pass that up?

This is one reason that building ASI is so attractive to such people in the first place: if they control a god-like being that enables them to become posthuman, there will never again be any risk of them losing their status. Mere humans will become the subalterns of the posthuman era. Revolt will become impossible.

Right from the start, Bostrom’s argument fails to get off the ground, because he naively assumes that the supposed benefits of an aligned ASI would be distributed among the masses in an egalitarian matter. Obviously, they won’t.

Benefitting All of Humanity?

You might rejoin that, while some of the AI companies are run by power-hungry sociopaths, most are explicit that their goal is to “benefit all of humanity” by building ASI and making its treasures available to everyone. OpenAI, Anthropic, etc. all make this claim.

The problem is that it’s an egregious lie that should be obvious to anyone who reflects for a nanosecond on the behavior of such companies thus far. The AI systems being developed right now — seen as the stepping stones to ASI — are built on massive intellectual property theft and the exploitation of workers in the Global South. Anthropic just paid out $1.5 billion in damages because they illegally pirated copyrighted material from shadow libraries like LibGen, and OpenAI hired a company that paid workers in Kenya as low as $1.32 an hour to label graphic images and descriptions of sexual assault, murder, etc. Some of these people ended up with PTSD.

Current AI models have a sizable environmental impact at exactly the moment when mitigating climate change matters most (e.g., Google’s emissions are up 51% due to AI, while Microsoft’s have risen by 30%), and they’re flooding the Internet with slop, disinformation, deepfakes, and other forms of digital pollution. People are experiencing AI-induced psychosis; some have committed suicide with the help of AI chatbots. xAI’s chatbot Grok spread literally millions of sexualized images on X, many involving underaged girls. A recent paper titled “How AI Destroys Institutions” provides a detailed, compelling argument for why AI threatens crucial civic institutions like the free press, the rule of law, and universities. This is a code red: AI poses an immediate threat to the stability of our world.

If these companies cared about “all of humanity,” they wouldn’t have done what they already did. ASI isn’t some wondrous threshold at which point such companies will suddenly start caring about whether they’re causing real harm to real people. If they don’t give a sh*t about us right now — about the trail of destruction they’re leaving in their wake — then why on Earth would anyone think they’ll care once they’ve built the most powerful technology in all of human history? A technology that could enable them to dramatically widen power disparities while simultaneously making such disparities permanent?

When most of humanity becomes economically obsolete and intellectually irrelevant in the post-ASI world, why would anyone even think the tech elite will keep us around? Creating an “aligned” ASI would constitute a death sentence for the majority of human beings on this planet, including you and me.

Again: if “aligned ASI” is even possible, if that’s even a coherent notion, it will be utilized by those who control it to further secure their positions atop the steep hierarchies of power, wealth, control, and domination. This truism completely demolishes Bostrom’s argument. In reality, the two options he should have highlighted are:

1. AI companies build an unaligned ASI that kills everyone on Earth, or

2. These companies build an aligned ASI that enables the rich and powerful to permanently entrench and heighten their position in society (while everyone else becomes obsolete).

The Worst People in the World

Even more, Bostrom fails to think deeply about the implications of mass immortality in a post-ASI world. “You get to live forever” sounds nice until you realize that so do the worst dictators, autocrats, fascists, and genociders. Bostrom tells us that we should accelerate toward ASI, but if the AI companies build ASI in the next three years, Trump may very well never die (again, going along with Bostrom’s dumb assumptions). Jared Kushner himself has stated that his generation may be the first to avoid death, so this is definitely on the radar of far-right American fascists:

Does Bostrom say anything about this? No. He simply assumes that ASI will use its god-like intellect to magically “solve” all the problems in the world — as if there’s a solution to the fact that I don’t want to share the world with immortal bigots, fascists, and genociders.

This is the problem with techno-utopian thinking: one can casually wave-away messy complications by declaring that an AI God will figure it all out. Everything will be wonderful. Everyone will be perfectly happy. We don’t know how, but you just gotta have faith in the divine powers of the ASI deity. HAVE A LITTLE FAITH!

Eternal Torture, Anyone?

There’s yet another problem with Bostrom’s argument. He assumes that the worst-case outcome is total human annihilation — everyone being killed by a rogue ASI. But things could be a lot worse. Since we’re already in the la-la-land of magical AI gods with inscrutable powers, we must consider the possibility of a misaligned ASI choosing not to kill us. Instead, it develops radical life-extension technologies that enable it to torture everyone on Earth until the heat death of the universe some 10^100 years from now. A literal hell in this world rather than the otherworld; in this life rather than the afterlife.

As soon as one recognizes this fantastical possibility — though no less fantastical than the ASI bringing about a “solved world” of immortality, perfect happiness, and endless cosmic delights — Bostrom’s calculations crumble. Indeed, the possibility of everlasting torture suggests that we should not just delay the onset of ASI, but take steps to ensure that it’s never ever built. The risk isn’t that accelerating ASI will get us killed next year, but that it catapults us into a state of perpetual misery and agonizing pain beyond human comprehension. Surely, then, it seems sagacious to delay the creation of such a being until we’re virtually certain that this outcome won’t obtain, no?

Does this ever cross Bostrom’s mind? Of course not. If it had, he would have realized that his argument doesn’t go through.

Extraordinary Hubris

Yet another problem is Bostrom’s failure to realize that there’s a big difference between dying and being murdered. The latter is worse than the former! I’d rather die at the age of 80 than be murdered next year. It is, therefore, morally appalling for someone to suggest that AI companies should plow ahead with ASI capabilities research even if there’s a ~97% chance of total annihilation in the near future. This is one of many points in the article when I wanted to shout: “F*ck you.”

Another concerns this: Who the hell is Bostrom — or the tech billionaires — to decide for everyone else what level of murder risk we should be exposed to? This is despicable. I, personally, don’t want to live forever. Yet I’m supposed to be okay with a ~97% chance of being a murder victim next year so that Bostrom and his transhumanist fanatic friends can maybe possibly live forever? (But again, it’s only the elite who will actually have access to immortality, for obvious reasons). Once more: “F*ck right off.”

None of this even makes sense given that transhumanists like Bostrom have signed up with cryonics companies to have their heads cryogenically frozen if they die before ASI arrives. Bostrom himself is a customer of Alcor.

Perhaps Bostrom has become skeptical that cryonics — a pseudoscience, as critics have pointed out for decades — could actually work. But if an aligned ASI will have magical powers, as Bostrom believes, then it will find a way to resurrect frozen corpses from the vat. There’s no problem too difficult for an ASI god to figure out!

This yields a very strong argument, from a specifically transhumanist perspective, for drastically slowing down capabilities research. If it takes 200 years to build an aligned ASI, then so what? Bostrom’s head will be cryogenically frozen for those two centuries. Whether an aligned ASI is built within his lifetime or 200 years after his untimely death, Bostrom gets to live forever. In contrast, if we race ahead and built a misaligned ASI that annihilates everyone, eternal life will become eternally out of reach for him. So, what’s the rush?

Speaking for Others

This is why Bostrom is forced to argue that immortality will be granted to all of humanity (which it won’t). Astonishingly, he argues that people in poor countries should be even more willing to accept a ~97% chance of being murdered in the future: they have even less to lose, because their existence is already so deplorable.

The arrogance here is striking. A super-privileged white dude from Sweden thinks he’s entitled to speak for “those who are old, sick, poor, downtrodden, miserable” (quoting him). The lived experiences of these people, their actual views and preferences, don’t need to be empirically investigated — not by someone as “smart” and “rational” as Bostrom. He can instead simply assume that they’d be okay with risking a brutal death-by-ASI in the coming years for a tiny chance of living forever.

This is the pernicious problem with the myth of objective rationality: it provides a kind of “license” for privileged people in their ivory towers to believe that they can know everything from all possible perspectives. But would people in the Global South actually be in favor of accelerating ASI research? I won’t speak for them, and neither should Bostrom.

A Flawed Analogy

Parts of Bostrom’s paper are also quite comical — such as when he uses the word “non-mundane,” which I guess is meant to sound smart? At another point, he calculates that if there’s a 20% chance of annihilation, and if AI safety research is moving slowly, then we should delay the creation of ASI by exactly 3.1 days. It’s hard to take this seriously:

Elsewhere, he warns that trying to pause or slow down AI capabilities research could backfire. One possibility he outlines is this:

To enforce a pause, a strong control apparatus is created. The future shifts in a more totalitarian direction. … The enforcement regime itself might also present some risk of eventually leading towards some sort of global totalitarian system.

As I mentioned at the beginning of this article, Bostrom has literally argued that we should seriously consider establishing a global surveillance system that monitors every person’s actions and utterances. Out of nowhere, he’s suddenly concerned that pausing AI might push us “in a more totalitarian direction”?

Furthermore, Bostrom insists that “the appropriate analogy for the development of superintelligence is not Russian roulette but surgery for a serious condition that would be fatal if left untreated.”

This is patently ridiculous; the analogy doesn’t work. If we build an unaligned ASI, then humanity will perish. But if we never build ASI, then humanity will persist. Our species is not facing a fatal illness of any sort! We could survive on Earth for the next 1 billion years, and then move to a new planet or solar system after that. In theory, we could survive until the heat death of the universe.

Yes, individual people will die, one at a time. But that’s very different from humanity as a whole dying in a single catastrophe. Unlike the former, the latter would entail the permanent termination of many things we care about, such as cultures, traditions, science, music, poetry, literature, philosophy, friendship, love, kindness, and there being future generations to carry on such things. The “common world,” as Hannah Arendt put it, would persist. If we were to go extinct, however, the common world and everything it contains — the cultural, intellectual, and artistic inheritance passed down from one generation to the next — would be lost forever.

A similar point could be made about Bostrom’s statement that “if nobody builds it, everyone dies.” Yes, every person alive today will die. But if we build a misaligned ASI, then every person alive today will be murdered and valued things like friendship, love, knowledge, cultures, etc. will also disappear. It’s hard to believe that “Swedish-superbrain Bostrom” doesn’t get this. Perhaps he would if he’d thought a bit more carefully about the matter.

Finally, it’s worth noting that one of Bostrom’s foils in the article is Roman Yampolskiy. Bostrom cites a 2023 Nautilus article coauthored by Yampolskiy, titled “Building Superintelligence Is Riskier Than Russian Roulette.” This is why he references “Russian roulette” in the analogy above. But did Bostrom actually read Yampolskiy’s paper? Is he unfamiliar with Yampolskiy’s work?

Yampolskiy argues that ASI alignment is fundamentally impossible. Why? Because advanced AI systems will have access to their own code. They will, consequently, be perpetually evolving over time. There is no static “ASI” to align to our values, once and for all. ASI will be a moving target, so to speak, and there’s absolutely no reason to believe that an aligned ASI at time T1 will remain aligned at T2.

This argument makes sense to me. It’s why I said above that I don’t believe in “AI alignment.” Yet Bostrom conveniently ignores Yampolskiy’s central thesis, which is what leads Yampolskiy to claim that we must (as far as I understand his position) permanently ban the development of ASI. There’s no way to guarantee that ASI will ever be “safe” for more than a moment. One iteration might be safe, but the next won’t.

Subscribe now

Conclusion

I don’t know how else to say it, but this was one of the dumbest papers I’ve ever read, from arguably the most influential TESCREAList of the past two decades.

Bostrom is completely disconnected from the messy reality of our world. He imagines that an aligned ASI will enable everyone to live forever, and that poor people should be especially willing to risk a ~97% chance of being murdered for a shot at immortality (which, we should note, is incompatible with many religious beliefs about the afterlife, meaning that many religious people would likely reject immortality). Bostrom fails to consider the implications of mass immortality, and doesn’t seem to realize that total annihilation isn’t the worst-possible outcome of building a Digital Deity with magical powers.

Everything about this article is drenched in arrogance and moral callousness. In other words, it’s a solid contribution to the TESCREAL literature! I’d recommend reading it if only because it offers a marvelous example of why I have nothing good to say about Bostrom as a person or scholar.

What do you think? Did you notice anything that I missed? As always:

Thanks for reading and I’ll see you on the other side!

“I’d like to die on Mars, just not on impact” — Elon Musk, who will die on Earth.

We begin with some chuckleworthy news from the world of space-expansionist billionaires: Elon Musk now says that SpaceX isn’t aiming to establish a colony on Mars in the near future, but will instead focus on building a lunar city. He writes:

For those unaware, SpaceX has already shifted focus to building a self-growing city on the Moon, as we can potentially achieve that in less than 10 years, whereas Mars would take 20+ years.

Some context: in 2011, Musk told the Wall Street Journal that people would be living on Mars within 10 years. In 2014, he declared that “the first people could be taken to Mars in 10 to 12 years, I think it’s certainly possible for that to occur.” In 2016, he predicted that humans would arrive on Mars by 2021, and that “SpaceX would start sending rockets to Mars by 2018, followed by a new Mars mission every 26 months.” He added: “If things go according to plan, we should be able to launch people probably in 2024 with arrival in 2025.” And so on.

It’s astonishing that Musk has gotten so far by habitually lying. Imagine that you give me $1 million to write a book. I assure you that it will be finished later this year. By the end of the year, I’ve only written two pages. But, I tell you, the reason progress has been so slow is that I actually need more money to keep writing. So you give me another $1 million, and the cycle repeats until I’m a billionaire. No one would take me seriously — I’d rightly be seen as a contemptible schmuck who cheated you out of a large heap of money.

This is what Musk is doing, of course. He makes grand promises, which he then consistently fails to keep. Thus far, “SpaceX has received at least $1 billion in government contracts, loans, subsidies and tax credits each year since 2016, and between $2 billion and $4 billion a year from 2021 to 2024 – while Tesla has received over $1 billion a year since 2020.” That’s taxpayer money. Our money, transferred from our pockets to his.

I want to scream what Jesse screams about Walter White in Breaking Bad (lol):

Doomsday Clock Setting

It turns out that I was wrong about the Doomsday Clock — at least partly. In a previous newsletter, I guessed that (a) the minute hand would move forward, and (b) it would be set to 88 seconds before midnight/doom — an apropos number given that “88” is code for “Heil Hitler” among white supremacists.

Instead, the Bulletin of the Atomic Scientists set their iconic clock to 85 seconds, which is 4 seconds ahead of its previous setting. This is the closest we’ve ever been to doom, and I’m afraid that I agree with the Bulletin’s dismal assessment. We are pretty much superfucked right now.

The 2026 Doomsday Clock announcement opens with this chilling paragraph:

A year ago, we warned that the world was perilously close to global disaster and that any delay in reversing course increased the probability of catastrophe. Rather than heed this warning, Russia, China, the United States, and other major countries have instead become increasingly aggressive, adversarial, and nationalistic. Hard-won global understandings are collapsing, accelerating a winner-takes-all great power competition and undermining the international cooperation critical to reducing the risks of nuclear war, climate change, the misuse of biotechnology, the potential threat of artificial intelligence, and other apocalyptic dangers. Far too many leaders have grown complacent and indifferent, in many cases adopting rhetoric and policies that accelerate rather than mitigate these existential risks. Because of this failure of leadership, the Bulletin of the Atomic Scientists Science and Security Board today sets the Doomsday Clock at 85 seconds to midnight, the closest it has ever been to catastrophe.

It’s stunning to watch our civilization sleepwalk toward the precipice, in complete denial of the extreme danger that we’re in. Nationalism, fascism, trade wars, and a general breakdown of the postwar global order are tearing down barriers to global conflict. Our political leaders behave like climate change doesn’t pose a dire, unprecedented threat to our collective wellbeing. AI slop, deepfakes, and disinformation spread through social media are flooding our information ecosystems with noxious poison, fomenting political polarization and conspiracy theories. And the US economy is propped up by an AI bubble that appears close to bursting, with potentially devastating consequences.

Meanwhile, the Trump administration allowed the New START nuclear arms treaty to expire last Thursday. This treaty was “the last major guardrail constraining the nuclear arsenals of the two countries that together hold some 85% of the world’s warheads.” Although there will be negotiations over the next 6 months between Russia and the US to establish a new agreement, there’s no guarantee that such an agreement will be reached.

This comes at a time when, as the Bulletin points out, “the Russia–Ukraine war has featured novel and potentially destabilizing military tactics and Russian allusions to nuclear weapons use,” while Trump has suggested that the US might restart underground nuclear weapons testing.

Interestingly, the Doomsday Clock announcement also mentions that,

in December 2024, scientists from nine countries announced the recognition of a potentially existential threat to all life on Earth: the laboratory synthesis of so-called “mirror life.” Those scientists urged that mirror bacteria and other mirror cells — composed of chemically-synthesized molecules that are mirror-images of those found on Earth, much as a left hand mirrors a right hand — not be created, because a self-replicating mirror cell could plausibly evade normal controls on growth, spread throughout all ecosystems, and eventually cause the widespread death of humans, other animals, and plants, potentially disrupting all life on Earth. So far, however, the international community has not arrived at a plan to address this risk.

I’ll admit that mirror life really freaks me out. It’s the stuff of nightmares: mirror bacteria could potentially infect our bodies without even being detected by our immune systems. It could also obliterate entire ecosystems, as the Bulletin notes, thus quite plausibly precipitating the complete extinction of our species.

As Dario Amodei — a guy I wouldn’t normally cite — points out in his surprisingly decent essay “The Adolescence of Technology”:

Skeptics retreated to the objection that LLMs weren’t end-to-end useful, and couldn’t help with bioweapons acquisition as opposed to just providing theoretical information. As of mid-2025, our measurements show that LLMs may already be providing substantial uplift in several relevant areas, perhaps doubling or tripling the likelihood of success. … We believe that models are likely now approaching the point where, without safeguards, they could be useful in enabling someone with a STEM degree but not specifically a biology degree to go through the whole process of producing a bioweapon.

I have no idea if AI could enable someone with the relevant training to synthesize mirror lifeforms. But that thought is not patently preposterous, which is alarming. And for those who suspect that there aren’t malicious actors in the world who would willingly try to destroy humanity, I published two academic articles several years ago (here) showing that, in fact, there are lots of people who would push a “doomsday button” — perhaps in the form of laboratory-synthesized mirror life — if only they could. I also wrote about this in a previous newsletter article.

Someone who held a senior position in the Biden administration once told me that he knew of two graduate students at Duke University who specifically pursued PhDs in microbiology because they wanted to synthesize a doomsday pathogen. Both were negative utilitarians, meaning that they believed our only moral obligation is to minimize suffering. Since the best way to do this is to eliminate that which suffers, negative utilitarianism instructs adherents to become, as R. N. Smart put it in 1958, a “benevolent world-exploder.”

(Of note: the Efilist movement — that’s “life” spelled backwards — embraces the negative utilitarian ethic, and one of its members bombed a fertility clinic in California last year in hopes of minimizing suffering by preventing new people from being born. I have no doubt that mirror life is on the radar of Efilists, as well as the possibility of using AI to build bioweapons.)

In conclusion, the world is in bad shape, and things are getting worse. That’s why the Doomsday Clock ticked forward. None of this is inevitable, though it often feels that way, as we’re up against forces that appear inexorable: billionaire techno-feudalists, TESCREAL fanatics driving the AGI race, fascists and authoritarians controlling world superpowers, and so on.

I’d recommend reading the entire Doomsday Clock announcement, which is quite short.

How TESCREAL Are YOU?

The other day, Timnit Gebru sent me a link to a survey that attempts to assess how TESCREAL one is. It’s quite amusing:

I took the 70-question comprehensive survey (less than 10 minutes), and here are my results. I’m surprised I scored this high!

Vitalik Buterin, who won a Thiel Fellowship and cofounded Etherium, also shared his score on Bluesky. Buterin is a major funder of the TESCREAL movement — e.g., he recently donated $660 million to the TESCREAL-aligned Future of Life Institute.

Someone else wrote this in response to the original post sharing the survey:

Those comments dovetail with my previous newsletter article. Indeed, it points to an idea that I haven’t yet written about (although I plan to in my book): Silicon Valley pro-extinctionists tend to see humanity, and life itself, different from human preservationists like myself. Consider this excerpt from an excellent article by my friend Dan Zimmer:

Today’s growing political fault line runs deeper than merely a preference for technological or ecological solutions, however, for it turns out that these new servants of Life also disagree on what Life itself is. One camp views Life primarily as an information process to expand and enhance, while the other conceives of Life chiefly as a complex system to maintain and balance. These contrasting perspectives inspire rival political visions: one gazing upward toward Life’s cosmic conquests and the other downward toward Life’s planetary entanglements.

TESCREALists tend to see life, and hence humanity, as entirely reducible to information processing. They see our brains as biological computers (“wetware”) performing computations in accordance with cognitive algorithms. Your personality, preferences, values, hopes, dreams, and conscious experiences are all just patterns of information being shuffled about by the wriggling neurons in your brain — that’s it.

When one accepts this view, it becomes immediately apparent that our biological substrate — our bodies and brains — aren’t important. If the same cognitive algorithms that give rise to your personality, values, thoughts, etc. were replicated on computer hardware, you would persist. More generally, if our intelligence is nothing but software, then we could presumably build artificial systems running even “better” software.

Many TESCREALists maintain that what’s valuable about humans is our intelligence, consciousness, and values. Since these don’t require the biological substrate of our bodies and brains to persist, there’s no reason to keep our species around. Simply transfer those attributes to computer hardware and then expand, augment, and enhance them. What matters, on this account, is preserving the abstract informational patterns that constitute life, intelligence, and consciousness. Ray Kurzweil himself calls this “patternism.” As he writes: “Our ultimate reality is our pattern.”

I wholly reject this view, which is why I so vigorously argue that preserving our biological species is important (at least for the foreseeable future, by which I mean many centuries or millennia). I reject it for a multitude of reasons, which I’ll save for another article. The point for now is to briefly highlight the more fundamental, metaphysical disagreements between people in my camp and the TESCREALists, who see flesh-and-blood humanity as disposable. More on this soon!

Podcast Episodes!

Finally, I keep forgetting to promote new episodes of my podcast Dystopia Now. Here is, once again, a recent discussion with my friend Dr. Alexander Thomas. I really enjoyed our chat, gloomy as it was, and think you might, too:

We also recorded a subscription-only episode on Stop AI, after one of it’s members defected and appeared to threaten violence against AI companies. This happened last November, and immediately afterwards, the individual in question disappeared. He is still missing as of this writing.

As always:

Thanks so much for reading and I’ll see you on the other side!

Given that some of the newly released Epstein files contain additional communications between Epstein and Chomsky, I’ve updated my article on the topic. It also turns out that Yudkowsky and Ben Goertzel are in the Epstein files, but I’ll save that for another article!

Acknowledgments: Thanks to Remmelt Ellen for providing critical feedback on an earlier draft. That does not imply that Ellen agrees with this post. For an excellent critique of the AI company Anthropic, written by Ellen, click here.

Our main topic today concerns the surprising similarities between Eliezer Yudkowsky and Ted Kaczynski, the Unabomber. As many of you know, Kaczynski was responsible for a campaign of domestic terrorism from 1978 until 1995, during which he sent home-made bombs through the mail to universities and airlines. This is why the media and FBI dubbed him the Un(iversity)-a(irline)-bomber.

In 1995, the FBI captured Kaczynski in his remote cabin near Lincoln, Montana, after Kaczynski coerced the Washington Post and New York Times into publishing a 35,000-word essay against aspects of technology. This is now called the “Unabomber Manifesto.” Kaczynski’s brother, David, recognized the writing style and reported him to authorities.

After pleading guilty to all charges, he was incarcerated in a supermax prison in Colorado, where he befriended the Oklahoma City bomber, Timothy McVeigh, whose actions Kaczynski described as “unnecessarily inhumane” (I’ll reference this again below). Kaczynski remained in prison until he committed suicide in 2023.

In his manifesto, Kaczynski rails against technology and industrialization. He also vociferously attacks leftism; he absolutely hated leftists! If the word had existed back then, he might have called himself an “anti-woke crusader.”

Some have interpreted Kaczynski as an “anarcho-primitivist,” or someone who wants humanity to return to the hunter-gatherer lifeways of our Pleistocene ancestors. But that’s not true: Kaczynski simply didn’t like industrial-era technology.

For example, he distinguished between “organization-dependent” and “small-scale” technologies: the former denotes “technology that depends on large-scale social organization.” This kind of technology cannot be reformed, he argues, given how interconnected it is. It comes as a kind of “unity,” meaning that “you can’t get rid of the ‘bad’ parts of technology and retain only the ‘good’ parts.” He gives the following example:

Progress in medical science depends on progress in chemistry, physics, biology, computer science and other fields. Advanced medical treatments require expensive, high-tech equipment that can be made available only by a technologically progressive, economically rich society. Clearly you can’t have much progress in medicine without the whole technological system and everything that goes with it.

In contrast, small-scale technology is that which “can be used by small-scale communities without outside assistance.” There are supposedly no examples of small-scale technologies being rolled back, precisely because they don’t rely on a vast network of other technologies to build and use. Since organization-dependent technology does — by definition — require such a network, sufficiently large shocks to the system can cause it to catastrophically crumble, thus leaving behind only the small-scale technologies that Kaczynski favors. For example, he writes that:

When the Roman Empire fell apart the Romans’ small-scale technology survived because any clever village craftsman could build, for instance, a water wheel, any skilled smith could make steel by Roman methods, and so forth. But the Romans’ organization-dependent technology DID regress. Their aqueducts fell into disrepair and were never rebuilt. Their techniques of road construction were lost. The Roman system of urban sanitation was forgotten, so that not until rather recent times did the sanitation of European cities equal that of Ancient Rome.

This is why reforming “the system” won’t work. We need a radical revolution, which he says “may or may not make use of violence; it may be sudden or it may be a relatively gradual process spanning a few decades.”

Kaczynski wasn’t much of an environmentalist. His primary motivation, on my reading, wasn’t to preserve the natural world, based on some notion of, e.g., biocentrism — a “value theory” that emerged in the 1970s, according to which nonhuman organisms possess intrinsic value (i.e., value in themselves) no less than humans. And he never advocated returning to the lifeways of our distant ancestors (“Back to the Pleistocene!”), as anarcho-primitivists do.

Rather, his primary concern was that the megatechnics of industrial civilization pose a dire threat to human freedom, dignity, and autonomy (his words). Hence, we need a revolution to completely decimate organization-dependent technology — the megatechnics of industrial society just mentioned — leaving behind only small-scale technologies.

Bill the Killjoy

Kaczynski’s manifesto, published with the promise that Kaczynski would desist his terrorist campaign, triggered a public debate about the consequences of advanced technology. Many people vehemently condemned Kaczynski’s homicidal actions while acknowledging that his manifesto made some compelling arguments.

One of these was a guy named Bill Joy, cofounder of Sun Microsystems who seems to have become a billionaire around 2000.1 On April 1 of that year (of all dates!), in the pages of Wired magazine (of all outlets!), Joy published a widely discussed article titled “Why the Future Doesn’t Need Us.” He writes that

From the moment I became involved in the creation of new technologies, their ethical dimensions have concerned me, but it was only in the autumn of 1998 that I became anxiously aware of how great are the dangers facing us in the 21st century. I can date the onset of my unease to the day I met Ray Kurzweil.

Kurzweil argued that the rate of technological “progress” is exponential, and hence that sentient robots are “a near-term possibility.” Joy had a personal conversation about this with Kurzweil at an event, and given how much Joy respected Kurzweil, he took this prognostication seriously.

Kurzweil then gave Joy a preprint copy of his forthcoming book The Age of Spiritual Machines, published in 1999. Joy writes that “I found myself most troubled by a passage detailing a dystopian scenario” in that book. This passage reads as follows (it’s a bit long, but worth reading in full):

First let us postulate that the computer scientists succeed in developing intelligent machines that can do all things better than human beings can do them. In that case presumably all work will be done by vast, highly organized systems of machines and no human effort will be necessary. Either of two cases might occur. The machines might be permitted to make all of their own decisions without human oversight, or else human control over the machines might be retained.

If the machines are permitted to make all their own decisions, we can’t make any conjectures as to the results, because it is impossible to guess how such machines might behave. We only point out that the fate of the human race would be at the mercy of the machines. It might be argued that the human race would never be foolish enough to hand over all the power to the machines. But we are suggesting neither that the human race would voluntarily turn power over to the machines nor that the machines would willfully seize power. What we do suggest is that the human race might easily permit itself to drift into a position of such dependence on the machines that it would have no practical choice but to accept all of the machines’ decisions. As society and the problems that face it become more and more complex and machines become more and more intelligent, people will let machines make more of their decisions for them, simply because machine-made decisions will bring better results than man-made ones. Eventually a stage may be reached at which the decisions necessary to keep the system running will be so complex that human beings will be incapable of making them intelligently. At that stage the machines will be in effective control. People won’t be able to just turn the machines off, because they will be so dependent on them that turning them off would amount to suicide.

On the other hand it is possible that human control over the machines may be retained. In that case the average man may have control over certain private machines of his own, such as his car or his personal computer, but control over large systems of machines will be in the hands of a tiny elite — just as it is today, but with two differences. Due to improved techniques the elite will have greater control over the masses; and because human work will no longer be necessary the masses will be superfluous, a useless burden on the system. If the elite is ruthless they may simply decide to exterminate the mass of humanity. If they are humane they may use propaganda or other psychological or biological techniques to reduce the birth rate until the mass of humanity becomes extinct, leaving the world to the elite. Or, if the elite consists of soft-hearted liberals, they may decide to play the role of good shepherds to the rest of the human race. They will see to it that everyone’s physical needs are satisfied, that all children are raised under psychologically hygienic conditions, that everyone has a wholesome hobby to keep him busy, and that anyone who may become dissatisfied undergoes “treatment” to cure his “problem.” Of course, life will be so purposeless that people will have to be biologically or psychologically engineered either to remove their need for the power process or make them “sublimate” their drive for power into some harmless hobby. These engineered human beings may be happy in such a society, but they will most certainly not be free. They will have been reduced to the status of domestic animals.

In Kurzweil’s book, you don’t find out until the next page that this is, in fact, an excerpt from the Unabomber Manifesto! Joy was shocked — as were most readers. This AI dystopia scenario actually seemed plausible, and perhaps something that could become our reality in the coming decades, given the exponential rate of technological development.

Kurzweil himself acknowledges that dystopia — a version of “doom” — is a very real possibility. He writes that “I was surprised how much of Kaczynskiʹs manifesto I agreed with,” as it constitutes a “persuasive … exposition on the psychological alienation, social dislocation, environmental injury, and other injuries and perils of the technological age.”

However, Kurzweil disagreed with Kaczynski that technologization and industrialization have been overall net negative — i.e., that there’s more bad than good associated with this transformation. Furthermore, he writes that

although [Kaczynski] makes a compelling case for the dangers and damages that have accompanied industrialization his proposed vision is neither compelling nor feasible. After all, there is too little nature left to return to, and there are too many human beings. For better or worse, we’re stuck with technology.

Subscribe now

Transhumanists Vs. Luddites

This brings us to a very important development that crucially shaped the TESCREAL movement, as well as the ongoing race to build ASI, or artificial superintelligence.

Note that the ASI race emerged out of the TESCREAL movement in the 2010s, thanks to the 2010 Singularity Summit, which enabled Demis Hassabis — cofounder of DeepMind — to solicit funding from Thiel. That got the first major company with the explicit goal of creating ASI off the ground.

The situation was this:

Everyone — Kurzweil, Joy, and Kaczynski, as well as a young Yudkowsky — agreed that the dangers of emerging technology are unprecedented and profound. But not everyone agreed about how we should respond to this apparent fact.

Joy argued that the only way to avoid a dystopian future like the one Kaczynski outlined is to impose strict moratoriums on entire domains of emerging science and technology through some sort of global treaty — perhaps on the model of “the 1972 Biological Weapons Convention (BWC) and the 1993 Chemical Weapons Convention (CWC).” He identified three problematic domains in particular: genetic engineering, molecular nanotechnology, and artificial intelligence, which he referred to using the acronym “GNR,” for “genetics, nanotech, and robotics.” In other words, Joy’s preferred response was “broad relinquishment” — basically, shut it all down, now!

This is very similar to Kaczynski’s revolutionary proposal: to broadly relinquish organization-dependent technologies to save humanity from a dangerous future in which we might lose everything that matters to us.

In contrast, Kurzweil and Yudkowsky were diehard transhumanists who claimed that, if GNR technologies are developed the right way, they could usher in a veritable utopia of posthuman immortality, superintelligence, and perfection. Once the Singularity happens, we will fully merge with machines and initiate a colonization explosion that floods the universe with the “light of consciousness,” such that the universe itself “wakes up,” to quote Kurzweil. This means that failing to develop GNR technologies is not an option. It would be tantamount to giving up on utopia forever, which is unacceptable!

Consequently, these transhumanists proposed an alternative response to what everyone agreed about: that GNR technologies are existentially risky. Instead of relinquishing entire domains of science and technology, we should instead found a new field dedicated to studying the associated risks with the goal of neutralizing them — so we can keep our technological cake and eat it, too.

This is how the field of Existential Risk Studies started. In 2001, the arch-transhumanist Nick Bostrom wrote a paper introducing the concept of an “existential risk,” which refers to any event that would prevent us from successfully transitioning to a utopian posthuman world. (This paper was then published in 2002.) The study of existential risks is thus an effort to ensure that GNR technologies usher in paradise rather than universal annihilation.

I myself was one of the most prolific contributors to the existential risk literature, and this is exactly how I understood the situation. I had read Joy’s article and Kurzweil’s books, and came to agree with Kurzweil that ASI is inevitable — someone somewhere at some point is going to develop it — so broad relinquishment is infeasible. It just won’t work. Rather, we need to buckle in and carefully examine the nature and etiology of existential risks, utilizing whatever tools we have to reduce the probability of an existential catastrophe as we push forward toward utopia.

Shut It All Down!

In a 2008 article, which I believe was written a few years earlier, Yudkowsky writes the following about the goals of his Singularity Institute (now MIRI):

Concern about the risks of future AI technology has led some commentators, such as Sun co-founder Bill Joy, to suggest the global regulation and restriction of such technologies. However, appropriately designed AI could offer similarly enormous benefits.

An AI smarter than humans could help us eradicate diseases, avert long-term nuclear risks, and live richer, more meaningful lives. Further, the prospect of those benefits along with the competitive advantages from AI would make a restrictive global treaty difficult to enforce.

The Singularity Institute’s primary approach to reducing AI risks has thus been to promote the development of AI with benevolent motivations that are reliably stable under self-improvement, what we call “Friendly AI.”2

Once again: utopia lies on the horizon, but we can’t get there without GNR technologies — especially ASI. So, what should we do? Establish organizations like the Singularity Institute to figure out how to make an ASI that doesn’t annihilate us but instead leads us to utopia.

From 2002 (when Bostrom founded Existential Risk Studies) until recently, Yudkowsky and his Rationalist-cult colleagues had been busy working away toward a solution to this problem, sometimes called the “value-alignment problem.” This is the central task of “AI safety,” which is a direct offshoot of Existential Risk Studies. Put differently, it’s the branch or subfield of Existential Risk Studies specifically focused on ASI.

However, one failure after another has led Yudkowsky and his doomer buddies to conclude that the value-alignment problem may take decades to solve — perhaps even centuries. Meanwhile, AI capabilities research — the effort to actually build an ASI, in contrast to AI safety’s goal of ensuring that the ASI is value-aligned — continues at something like an exponential pace. Or so they claim.

This leaves us in an immensely dangerous situation: companies like DeepMind might build an ASI before we know how to control it. And if we can’t control it, then the default outcome will be doom for everyone on Earth.

Yudkowsky Comes Full Circle

Consequently, Yudkowsky has started to adopt a position strikingly similar to Joy’s: the only feasible option right now is to shut it all down, as Yudkowsky argued in a 2023 article in TIME magazine. Whereas in 2008 he was claiming that “a restrictive global treaty” would be “difficult to enforce” — it is, essentially, a nonstarter — by 2023 he’s explicitly arguing for a global treaty to prevent any company anywhere on Earth from developing ASI within the foreseeable future.3

This is essentially a broad relinquishment proposal because Yudkowsky is saying that the entire field of AI capabilities research must be halted immediately and indefinitely. With another several decades, or perhaps a few centuries, AI safety research can finally catch up to capabilities research, at which point we should proceed to build ASI — but only once we’re extremely sure that the outcome will be utopia rather than doom.

There are also echoes of Kaczynski here. As my friend Remmelt Ellen points out (personal communication),

both Kaczynski and Yudkowsky want to achieve ultimate freedom – and to not be inhibited by what those (normie) feminists had to say. Both are libertarians with anti-woke inclinations. They just ended up with radically different conceptions of what “being free” versus “being locked up” means. For Kaczynski, it was being caged by the technological system. For Yudkowsky and other transhumanists, it’s being caged in a mortal suboptimal human body (not having the right kind of technological innovation).

Remmelt adds that both are also “white privileged guys who think they can play with the lives of others for their idealized freedoms.”

Here’s what I would add: Kaczynski saw humanity as confronting an apocalyptic moment in which the only option to save humanity from existentially disastrous technology is to overthrow the reigning paradigm — industrial society. Yudkowsky similarly argues that we’re in an apocalyptic moment where the only option left is to overthrow the current configuration of our technological milieu, whereby AI capabilities research is racing toward the ASI finish line much faster than AI safety research. Something radical and drastic has to change, now.

To be clear, Yudkowsky isn’t calling for the overthrow of our entire industrial system. But he is advocating for a major system within that system to be fully and indefinitely dismantled. Consider that the ASI race is backed by more than $1.5 trillion, as of 2025. It’s led by some of the most powerful tech billionaires on the planet — many of whom have direct links to the most powerful political figures in the world like Trump and JD Vance. Such figures will vigorously resist any deceleration of the ASI race, in part because they think we’re in an AI game of chicken with our geopolitical rival, China. Furthermore, this race has become a sizable portion of the US economy — “a little under 40% of average real GDP growth” during 2025, according to CNBC — and if the “AI bubble” were to burst, it could plunge the entire economy into a great recession, or even depression.

An immediate stop to the ASI race would require a fundamental shift away from the currently entrenched social, political, and economic system-state. It would amount to a radical pivot from the status quo. Yet, if we fail to do this in the coming months or years, quite literally everyone on Earth will die, Yudkowsky claims.4

Subscribe now

“Solution: Be Ted Kaczynski”

There are even more striking similarities. These relate to the strategies or tactics that might be employed to save humanity from technology gone awry. In the aforementioned TIME article, Yudkowsky argues that states should be willing to engage in targeted military strikes against “rogue data centers” to prevent ASI from being built in the near future. If anyone violates this global treaty, we should respond as if it’s an existential threat to one’s country.

He says militaries should do this even at the risk of triggering a thermonuclear war. When asked on social media, “How many people are allowed to die to prevent AGI” from being built near-term, he said basically everyone on Earth:

There should be enough survivors on Earth in close contact to form a viable reproductive population, with room to spare, and they should have a sustainable food supply. So long as that’s true, there’s still a chance of reaching the stars someday.

Since the minimum viable human population may be as low as 150 people, that means that roughly 8.2 billion people are “allowed” to die to protect the utopian fantasies that a value-aligned ASI could, they claim, realize.

On another occasion, he was asked about “bombing the Wuhan center studying pathogens” to prevent the Covid-19 pandemic, to which he responded that it’s a “great question” and “if I can do it secretly, I probably do and then throw up a lot.” This makes one wonder whether he’d be willing to bomb AI laboratories to prevent an ASI catastrophe — an outcome almost infinitely worse, from the TESCREAL perspective, than the pandemic.

He’s endorsed property damage to prevent the creation of ASI, and once said that he’d be willing to sacrifice “all of humanity” to create god-like ASI that flit about the universe “to make the stars cities.”

That being said, the eschatology of Yudkowsky’s view is clearly different from that of Kaczynski’s. The latter wants a global return to communities built around small-scale technologies, whereas Yudkowsky wants cosmic-scale megatechnics by developing ASI the “right way.” Kaczynski saw our current path as a complete dead-end — hence, the need for a revolution that demolishes industrial society — whereas Yudkowsky imagines there being only one way forward: across a narrowing bridge of further technologization that might collapse at any moment if we’re not extremely careful. Yet Yudkowsky has spoken approvingly of strategies and tactics not that different from Kaczynski’s.

In fact, his endorsement of military strikes at the risk of starting a nearly extinctional thermonuclear war goes way beyond what Kaczynski would have ever endorsed. Recall Kaczynski’s comments about the domestic terrorism of McVeigh, which he described as “inhumane.” Although Yudkowsky hasn’t committed any acts of violence — and let’s hope that remains the case — he’s flirting with scenarios far more extreme than anything Kaczynski came close to approving.

Even more, Yudkowsky’s apocalyptic rhetoric and statements about risking nuclear war, bombing Wuhan, etc. have shifted the Overton window within the Rationalist cult. Consider an AI safety workshop at the end of 2022, shortly after OpenAI freaked out the Rationalists by releasing ChatGPT. This workshop was put together by three people in the TESCREAL movement, one of whom went on to work for Yudkowsky’s Thiel- and Epstein-funded organization MIRI. As I wrote in an article for Truthdig:

Although MIRI was not directly involved in the workshop, Yudkowsky reportedly attended a workshop afterparty.

Under the heading “produce aligned AI before unaligned [AI] kills everyone,” the meeting minutes indicate that someone suggested the following: “Solution: be Ted Kaczynski.” Later on, someone proposed the “strategy” of “start building bombs from your cabin in Montana,” where Kaczynski conducted his campaign of domestic terrorism, “and mail them to DeepMind and OpenAI lol.” This was followed a few sentences later by, “Strategy: We kill all AI researchers.”

Participants noted that if such proposals were enacted, they could be “harmful for AI governance,” presumably because of the reputational damage they might cause to the AI safety community. But they also implied that if all AI researchers are killed, this could mean that AGI doesn’t get built. And foregoing AGI, if properly aligned, would mean that we “lose a lot of potential value of good things.”

Be Ted Kaczynski and start sending bombs from your Montana cabin to save humanity! The circle back to Kaczynski — going beyond Joy’s non-violent broad relinquishment proposal — is complete. As Remmelt writes:

While Yudkowsky has not performed actions like Ted, he is also an ideologue pushing for his notion of ultimate freedom that plays with the lives of people. And Eliezer’s past rhetoric risks inspiring Kaczynski-style actors. What started with Ted Kaczynski ended with Yudkowsky inspiring Kaczynski-style thinking.

Dario Amodei and Bill Joy

This is a fascinating narrative arc, in my opinion. Some of the very same transhumanists who scoffed at Joy’s proposal are now arguing that we must “shut it all down.” A subset of these transhumanists, including Yudkowsky, are increasingly discussing and thus normalizing talk of violence to achieve their aims of stopping AI capabilities research.

As it happens, in a rambling post on X, in which Yudkowsky bizarrely claims he never knowingly engaged in statutory rape [spits coffee], he also says he never read the Unabomber Manifesto. That may be true, but it doesn’t change the fact that Yudkowsky and his Rationalist cult share many notable similarities with Kaczynski and his radical brand of neo-Luddism.

It’s also worth noting that Dario Amodei, cofounder of Anthropic, recently published an article titled “The Adolescence of Technology” in which he references both Joy and Kaczynski. He says that “I originally read Joy’s essay 25 years ago, when it was written, and it had a profound impact on me,” adding that

then and now, I do see it as too pessimistic — I don’t think broad “relinquishment” of whole areas of technology, which Joy suggests, is the answer — but the issues it raises were surprisingly prescient, and Joy also writes with a deep sense of compassion and humanity that I admire.

Amodei doesn’t see relinquishment as an answer because he’s in the TESCREAL movement — specifically, a camp of TESCREALism that hasn’t come full-circle with Joy and Kaczynski, unlike Yudkowsky and his followers. Amodei still holds the view that Yudkowsky and other folks in Existential Risk Studies once embraced: we should continue apace while being dutifully cautious about the possibility of bad outcomes. Yudkowsky’s view is much more pessimistic, and hence more in-line with Joy’s: unless we shut it all down right now and reallocate our resources to studying existential risks, AI scientists are going to kill everyone.

Here you can see the continuing influence of Joy’s 2000 article — and Kaczynski’s manifesto.

Zooming out even more, it’s worth noting that the TESCREAL movement has fractured into competing camps over the past 5 years or so, since the release of ChatGPT: (1) the doomers, (2) the accelerationists, and (3) those somewhere in the middle. Yudkowsky, of course, exemplifies (1), whereas Kurzweil and Amodei fall into category (3). Kurzweil, for example, argues that there might be up to a 50% chance of annihilation due to GNR technologies, but that the only way forward is through. Amodei has also acknowledged that the technology he’s building might result in an existential catastrophe (his p(doom) is at 10-25%).

The odd man out is (2), as they strongly disagree with Kaczynski, Joy, Yudkowsky, Kurzweil, and Amodei that there’s any significant risk at all. Rather, they believe the default outcome is utopia — annihilation isn’t really on the menu of options. Because of this, they constitute a new position that wasn’t present during the Joy-Kurzweil debate: rather than agreeing that there are serious risks but disagreeing about what the right response is, they simply reject the claim that there are any serious risks at all.

Examples of accelerationists in category (2) are Marc Andreessen and Gill Verdon, better known online as “Beff Jezos.”

Conclusion

I hope you found this article interesting and, hopefully, somewhat insightful. I’d love to know what you think I might be missing, or wrong about, in the comments. As always:

Thanks for reading and I’ll see you on the other side!

Incidentally, Kate and I recently recorded a podcast episode on this very topic, which you can listen to here.

What follows is quite long! Despite being way more technical than what I usually publish here, this aims to be a comprehensive yet relatively accessible introduction to quantum mechanics, quantum computers, and the potential uses of such computers. You can think of it as a mini-book consisting of three chapters: 1, 2, and 3.

Topics discussed include:

• Quantum superposition

• Quantum interference

• Quantum entanglement

• Nonlocality

• Qubits

• Quantum parallelism

• Quantum algorithms

• Computability

• Computational complexity theory

• Quantum error correction

• Decoherence

• Shoes

• Tennis balls

I really hope you find this interesting — I’ll return to my usual kvetching about TESCREALism and the reckless race to build AGI next week, probably with an article titled something like: “Ted Kaczynski and Eliezer Yudkowsky: Two Peas in a Pod,” which will explore the surprising links between Kaczynski’s radical neo-Luddism and contemporary AI doomerism of the sort championed by Yudkowsky. Fun stuff!

Without further delay, here’s this week’s lengthy article!

Chapter 1: Quantum Mechanics: The Basics!

Understanding quantum computing requires some background knowledge of quantum phenomena relating to superposition, interference, entanglement, and measurement. Let’s consider these in turn.

What makes quantum computers (and quantum computation) different from so-called “classical” computers (and “classical” computation) — that is, the sorts of computers and computation that are ubiquitous today — is that quantum computers exploit the perplexing properties of quantum mechanical phenomena. Consider the fact that all matter in the universe exhibits wave-like properties. This includes macroscopic objects like you and I, cars and cats, tennis balls and planets. However, the wavelength associated with an object decreases inversely with that object’s mass. The more massive the object, the smaller its associated wavelength. This is why macroscopic objects like you and I do not exhibit wave-like behavior. We behave, instead, like discrete entities in the world. This is what “classical” physics is very good at describing.

However, when one zooms-into the submicroscopic realm of protons, electrons, and photons — that is, particles of light, which have no mass, unlike protons and electrons — the wave-particle duality of these entities becomes very significant. Quantum computers exploit this fact of nature: very tiny things behave as both particles (discrete entities) and waves. These properties of quantum particles can then be manipulated in various ways to perform useful computations, such as simulating physics and solving certain computationally complex problems (more on this below).

An illustration of the wave-particle duality of quantum particles comes from the famous “double-slit” experiment. Let’s begin by imagining the experimental set-up using macroscopic objects like tennis balls (see figure 1), which we can describe as “classical” particles, since they obey the laws of classical physics—i.e., the physics of objects massive enough for their wavelengths to be irrelevant.

Consider a “gun” that shoots out tennis balls in the general direction of a barrier, which consists of two rectangular slits just wide enough for a tennis ball to pass through them. On the other side of this barrier is a wall covered in velcro, which the tennis balls can thus stick to. The gun that shoots out the tennis balls is not very accurate, so only some of the tennis balls end up passing through one of the two slits in the barrier.

Now, imagine turning on the gun and shooting tennis balls toward the barrier, one at a time, for several hours. What would we expect to see? A whole bunch of tennis balls would hit the barrier and bounce back toward you and the gun, but some of the tennis balls would pass through one of the two slits. You would then see two rectangular areas on the velcro wall covered in tennis balls.

There’s nothing mysterious about this scenario — it’s exactly what you would intuitively expect, because macroscopic objects like tennis balls behave as discrete entities rather than as waves. The ball passes through one or the other slit, and ends up on the velcro wall just behind that slit.

But because very small entities have properties of both particles and waves, this is not what occurs with entities like photons or electrons. If you replace the tennis balls with, say, electrons, and shoot them one at a time (just as you did with the tennis balls), you will find that the electrons cluster together on the wall behind the barrier in multiple locations (see figure 2). Between these locations where the electrons cluster together, there are gaps in which no electrons appear on the wall.

Why does this happen? Because, although the electrons are shot out of the gun one at a time — as “quanta” — they pass through the slits as waves. And when waves pass through two slits, they diffract around each slit. The diffracted waves that emerge on the other side of the barrier move away from the slits in a circular manner (see figure 3), which enables these waves to interfere with each other: in some cases, the peak of one wave, from one of the two slits, coincides with the peak of the wave from the other slit; in other cases, the peak of one wave, from one of the slits, coincides with the trough (or low-point) of the wave from the other slit. When the wave then hits the wall behind the barrier, it manifests itself as a single, discrete particle in a specific place on the wall.

Interference, Superposition, and Measurement

There are four important points to make about this.

First, when the peak of a wave coincides with the peak of another wave, the amplitude (or size) of the two waves is added together. Consequently, the amplitude of the resulting wave — i.e., the combination of these two waves added together — is twice as large. This is called “constructive interference.”

In contrast, when the peak of a wave coincides with the trough of another wave, the amplitudes of the two waves (assuming these amplitudes are equal, for the sake of simplicity) cancels out. Consequently, the amplitude of the resulting wave becomes zero, due to “destructive interference.” (See figures 4 and 5.) This is the way that all waves behave — e.g., if you were to perform the double slit experiment half-submerged in water, you would find the waves interfering on the other side of the barrier in exactly the same way.

Second, when the wave of an electron reaches the wall behind the barrier, it is, so to speak, “forced” to choose a specific location. At this point, it manifests as a particle. But before it manifests as a particle, it behaves like a wave, essentially passing through both slits simultaneously. Indeed, if it didn’t pass through both slits simultaneously in a wave-like manner, then there wouldn’t be the “interference” pattern on the wall behind the barrier, which can only result from something behaving like a wave. (If it only passed through one of the two slits, instead of both, then you’d see the exact same pattern as the tennis balls on the velcro wall.)

Physicists refer to this strange phenomenon as superposition: the electron, in a sense, is in two states at the same time — one state corresponding to “passing through the first slit,” and the other corresponding to “passing through the second slit.” Superposition is one reason that quantum computers could solve some problems exponentially faster than classical computers, as we’ll discuss more in chapter 2.

Third, although the electron, in a very perplexing way, passes through both slits simultaneously, it manifests as a single particle in a specific place on the wall behind the barrier. The reason is that the wall acts as a kind of measurement, and whenever quantum systems in superposition are measured, they always “collapse” into some particular state.

This is very different from our own experiences on the macroscopic level, where the wave-like behavior of macroscopic objects is negligible. To paraphrase one quantum computing expert, Scott Aaronson (see below), superposition is only something that quantum particles like to do in private. As soon as one looks to see where exactly the particle is, the wave-like behavior disappears and suddenly the particle manifests in some particular place. But where exactly does it appear?

This leads us to the last important point. The probability of finding a particle, such as the electron in our double-slit experiment, at a specific location is a function of the amplitudes of its wave-like behaviors. When the two waves that emerge from the double slits interfere constructively, the probability of finding an electron — if one were to measure it — is very high. When the two waves interfere destructively, the probability is very low (or quite literally zero).

Hence, the interference pattern of discrete electron-particles on the wall corresponds to the constructive and destructive interference of its waves. That is to say, the electrons appear mostly where the peaks of its two waves coincide, and they don’t appear where the peaks and troughs of its waves coincide. In other words, combine the amplitudes of the two waves and you get the probability of the electron actually appearing there if a measurement of the system is taken.

Unsolved Puzzles

The quantum phenomena of superposition and interference are crucial for understanding why and how quantum computers could be far superior to classical computers in certain respects. As the renowned physicist Richard Feynman once quipped, “the double-slit experiment has in it the heart of quantum mechanics,” to which he added, “in reality, it contains the only mystery.”

The mystery here is what exactly is going on at the quantum level: we know what happens — as described above — but we don’t really understand why. For example, consider that there are multiple locations on the wall, behind the barrier, where the waves from the single electron interfere constructively. The probability is thus high that the electron will appear in one of these locations. But why one location rather than another? Why in this place where the waves interfere constructively rather than that other place where they also interfere constructively? No one knows.

Another puzzle concerns the measurement of some system: why do particles such as the electron only want to be in a state of superposition in “private,” to borrow Aaronson’s wording from above? Why is it that measuring, or observing, a quantum system “forces” it to fall into some definite state? Again, we don’t have an answer, and there are many interpretations of what is happening when quantum systems fall into definite states (e.g., perhaps the universe splits into two timelines, a bizarre possibility called the “many-worlds interpretation” of quantum physics).

But scientists don’t need to have an answer to such questions to exploit such phenomena for the purposes of quantum computation. Understanding the way quantum systems behave, even if we don’t know why, is enough for quantum computing technology to work.

Quantum Entanglement

Yet another mysterious phenomenon relevant to quantum computers is called “entanglement.” Whereas the double-slit experiment involves only single particles (being shot out of the gun one at a time, where the particle’s own waves interfere with themselves as they pass through both slits simultaneously), entanglement involves two or more particles.

Two particles can become “entangled” in numerous ways — for example, by interacting with each other. Once entangled, the state of one particle cannot be described independently of the state of the other particle. Consider an example often used by physicists: imagine that I have a pair of shoes and two boxes. I put the left shoe in one box and send it to my friend Sarah, and I put the right shoe in the other box and send it to my friend Dan. I then tell Sarah and Dan exactly what I have done, but don’t reveal to them whether they will be receiving the left or right shoe.

If Sarah receives and opens her box first, she will immediately know that Dan received the box with a shoe for the right foot. That is to say, looking and seeing which shoe she received provides information not just about the shoe I sent her, but about the shoe that I sent Dan. In this way, you could say that the shoes are “entangled,” as the “state” of one box is not independent of the other. If there’s a left shoe in one box, then there must be a right shoe in the other box. There is nothing mysterious about this situation.

But there is something mysterious about the quantum analog of this example. Consider that particles like electrons have a property that physicists call “spin.” An electron could be “spinning” either “up” or “down.” Those are the two options, just as left and right are the two options for the aforementioned shoes. Now, if two electrons become entangled, they will always have opposite spins: if the first electron has an “up” spin, then the second electron will always have a “down” spin — and vice versa.

Now imagine that I take two entangled electrons and send one to Sarah and the other to Dan. If Sarah receives and examines — that is, measures — the spin of her electron to be “up,” she will immediately know that the spin of Dan’s electron is “down.” However, because of superposition, the electron that Sarah (and Dan!) received is neither up nor down until she examines — that is, measures — it. In other words, both electrons are in a superposition of “up” and “down,” and it is only when Sarah measures her electron that it is “forced” to choose between a definite “up” or “down” orientation. (Yes, this is really, really weird stuff!) Sarah could have thus found her electron with a “down” spin, in which case she would immediately know that Dan’s electron has an “up” spin.

This clearly contrasts with the shoe example, in which the shoe in Sarah’s box is already in a definite state: it is the left-foot shoe. Sarah simply doesn’t know this. If the shoe were in a quantum state of superposition, it would be neither left nor right until Sarah opens the box, thereby making a “measurement” of which foot (left or right) the shoe that she received is.

Non-Locality

This is extremely counterintuitive, once again. It is not how things work in our macroscopic world, and so we lack the intuitions necessary to fully comprehend what the mathematics, backed by empirical observations, unambiguously imply.

Even more mysterious is how the spins of the electrons are correlated: by “forcing” one of the electrons into an “up” or “down” state, the other electron is instantaneously “forced” into the opposite state. It is almost as if the first electron measured sends a signal — faster than the speed of light, because this occurs instantaneously — to the second electron saying: “Hey! I now have a definite ‘up’ orientation, so you should have a definite ‘down’ orientation.” These electrons could be literally billions of miles away from each other: as soon as one electron is measured, or observed, the other electron assumes a definite state that is opposite of the first.

This leads some to speculate that information could be sent over vast distances faster than the speed of light, via entanglement. However, transferring information via quantum entanglement faster than the speed of light is not possible, and indeed there is no information exchanged from the first electron measured to the second. That is to say, the first electron does not “communicate” its spin to the second electron.

Rather, what happens is that when two particles are entangled, they become part of the same “wavefunction.” A wavefunction describes how an isolated quantum system changes or evolves over time. With respect to the double-slit experiment, the wavefunction associated with each electron shot out of the gun toward the barrier and wall behind it, describes how the electron, behaving as a wave, passes through both slits, interferes with itself on the other side, and creates the interference pattern on the wall (corresponding to the probability of the electron appearing at any given location). That’s what the wavefunction does!

Hence, if two electrons become entangled such that they share the same wavefunction — where this wavefunction isn’t the two electron’s wavefunctions combined, but a distinct new wavefunction that describes the entire system consisting of these two electrons — then any measurement (or observation) made of one electron in the system will “force” the other electron in the same system into some definite state. And since the spin orientation of the second electron must be the opposite of the spin orientation of the first, measuring the first electron as having an “up” spin instantaneously causes the second electron to have a “down” spin—because, once again, they are part of the very same quantum system that one “forces” into a definite state by measuring or observing any part of that system.

This phenomenon is call “nonlocality,” whereby quantum systems that might not be localized in space (again: entangled particles could be separated from each other by literally billions of miles and still exhibit this property) are nonetheless instantaneously correlated. “Classical” physics cannot explain this, nor is there any analog in our own experiences of the macroscopic world, which is governed by “classical” physics.2

Conclusion

That does it for chapter 1. Congratulations on making it to the end of this short intro on quantum mechanics! We now turn to the ways that quantum computers exploit the phenomena of superposition, interference, entanglement, and measurement.

Chapter 2: The Potential Power and Limitations of Quantum Computers

In chapter 1, we discussed three important properties of quantum systems: superposition, interference, and entanglement, along with the phenomenon of measurement. These are foundational to quantum computing. They are the reason that quantum computers could be exponentially faster and more efficient than classical computers with respect to certain kinds of problems.

What the Heck Are Qubits?

Consider that classical computers encode information as bits, which take the form of either a 1 or 0. In contrast, quantum computers encode information as qubits, short for “quantum bit.” Unlike classical bits, qubits are measured as either 1s or 0s, but prior to being measured they exist in a superposition of both 1 and 0. This means that qubits can store far more information than bits.

For example, take the following sequence of eleven classical bits: 11111011011. How much information do these bits together store? The answer is only a single number. When translated from binary code, this sequence gives the number 2011. Compare this to a sequence of eleven qubits. Because of superposition, these qubits are neither 1s nor 0s, but exist in a superpositional state of (0+1) prior to being measured. Hence, listing (0+1) eleven times yields: (0+1)(0+1)(0+1)(0+1)(0+1)(0+1)(0+1)(0+1)(0+1)(0+1)(0+1). How much information can these qubits store? The answer is every single number from 1 to 2,048.

To underline this point, eleven classical bits stores just one number. Eleven qubits stores 1, 2, 3, 4, … all the way up to 2,048. That’s a huge difference! (See slide 5 of this presentation for citation.)

More generally, simulating some number n of qubits on a classical computer would require approximately 2^n computational steps, because n qubits can be in a superposition of 2^n states. Hence, as the number of qubits increases, the number of steps needed to match the computational power of quantum computers grows exponentially.

It would thus require 8 classical steps to simulate 3 qubits (2^3), and 16 classical steps to simulate 4 qubits (2^4). Now consider a quantum computer that contains 300 qubits. Simulating these qubits on a classical computer would require roughly 10^90 steps — since 2^300 roughly equals 10^90 — which is larger than the total number of atoms in the universe (estimated to be approximately 10^80). That is the extraordinary power of superposition.

Entanglement, then, is what enables quantum computers to perform many computations at once. This is to say, in a single operation, quantum computers can manipulate multiple values of the qubits at the same time, whereas classical computers must manipulate each value of the bits individually, one at a time.

For example, if two qubits are entangled — meaning that they are part of the same quantum system, described by a single wavefunction — and one is measured to have a value of 1, then the other qubit will immediately take the value of 0. In classical computing, it is not possible to create an algorithm where two bits have no value, but nonetheless have opposite values, though it is possible in quantum computing.

This enables a phenomenon called quantum parallelism, which contrasts with the way that computation happens in a classical microchip: serially rather than in parallel. Quantum parallelism is why quantum computers can perform calculations much faster than classical computers, including the most powerful classical computers: supercomputers.

Finally, interference is used in quantum computers to determine the correct solution(s) to a problem. When quantum particles are in superposition, the superimposed states of these particles produce wave-like phenomena that can interfere with itself, just as the waves of the electron in the double-slit experiment interfere with themselves to produce the interference pattern seen on the wall. One can thus design quantum algorithms (discussed in a bit) that manipulate the waves associated with particles in the hardware of quantum computers so that the positive interference of these waves corresponds to a correct answer, while negative interference corresponds to an incorrect answer.

For these reasons, quantum computers are not just “better” versions of classical computers. They are fundamentally different, in perhaps the same way that an incandescent light bulb isn’t just a better “version” of a candle — to borrow an analogy that others have previously made.

The Limitations of Quantum Computers

Quantum computers could revolutionize several billion-dollar industries and enable exciting new scientific discoveries. There is, however, a widespread belief that they will be superior to classical computers in nearly every way. This is not true: quantum computers will provide modest speed-ups with respect to certain problems, exponential speed-ups with respect to a few others, and no advantage in certain other cases.

In other words, classical computers will still out-perform quantum computers on particular tasks, as discussed more below. There is no fundamental difference in computability between classical and quantum computers — that is, quantum computers are not able to solve problems that classical computers cannot solve in principle.

The sole advantage of quantum over classical computers is their speed and efficiency at solving problems, including some that would take classical computers centuries, millennia, or longer to solve. That is to say, the exceptional speed and efficiency of quantum computers means that there may be some problems that they can efficiently solve that classical computers cannot, simply because classical computers require impractical quantities of resources to find a solution.

Computational Complexity Theory

To get a sense of the limitations and promise of quantum computers, consider figure 6. This shows three important classes of problems, each defined by their computational complexity, where “computational complexity” concerns the amount of resources an algorithm would need to solve the problems. The greater the computational complexity, the more resources are required — and vice versa.

Decision problems in the class of P are those that can be efficiently solved by a “deterministic Turing machine,” such as the classical computers that we use today.

• By “decision problems,” we simply mean those problems that, given some input, have a definite “yes” or “no” answer. Examples include: “Is Paris in Spain?” (no!!) and “Does 4 plus 4 equal 8?” (yes!!).

• By “efficiently,” we mean that the problem can be solved in polynomial time, where “polynomial time” refers to an amount of time that increases polynomially with the size of the input. The larger the input, the greater the amount of time needed to solve the corresponding problem, where the input and the time needed are related by a polynomial (a mathematical expression consisting of variables and coefficients).

All P problems are also NP problems, but not all NP problems are — it seems — P problems. In other words, P appears to be a subset of NP. The difference is that NP denotes the much larger class of problems that can be efficiently verified (or checked) in polynomial time. If a program can efficiently solve a decision problem, then it can also efficiently verify that problem, too. Why? Because verifying a solution will always be easier than solving the problem itself. Hence, there are some problems that can, it seems, be efficiently verified but not efficiently solved: these are the NP problems, whereas P problems are those that can be efficiently solved and verified by a deterministic Turing machine.

P = "Yes or no" problems that are efficiently solvable and efficiently verifiable by a deterministic Turing machine, like the computers we currently have.

NP = "Yes or no" problems that are efficiently verifiable but not efficiently solvable by a deterministic Turing machine.

The most notorious unsolved puzzle within the field of computational complexity is whether the class of P and NP problems is, in fact, coextensive: does P equal NP, or not? This is to say, are all the problems that are verifiable in polynomial time also solvable in polynomial time? Again, it seems like the answer is “no”: some problems can be efficiently verified but not solved. But we do not know this for sure, and there is a $1 million dollar prize awaiting anyone who can mathematically prove that P equals NP, or demonstrate the opposite.

The final class of computational complexity that’s worth mentioning goes by the name “NP-complete” — the most infamous of the NP problems. It includes any problem that, if efficiently solved by an algorithm, would enable one to solve all the problems in NP. This is a rather esoteric notion, but for our purposes the key point is that, as the philosophers Michael Cuffaro and Amit Hagar explain, “since the best known algorithm for any of these [NP-complete] problems is exponential, the widely believed conjecture is that there is no polynomial algorithm that can solve them” (italics added).

In other words, NP-complete problems appear to require superpolynomial, if not exponential time, to solve, meaning that as the size of the input increases, the amount of time required to solve an NP-complete problem grows at a superpolynomial or exponential rate. Hence, there are ostensibly no algorithms that could solve such problems efficiently.

An example of an NP-complete problem is the jigsaw puzzle: there is no algorithm that can efficiently solve the problem of fitting all the pieces together for any arbitrarily large puzzle because of the combinatorial explosion of different ways the puzzle’s pieces could fit together. (It may help to imagine here a jigsaw puzzle that has no picture on the front; all the pieces are blank.)

As the number of pieces in the puzzle increases, the problem of finding the correct solution thus becomes computationally intractable — due to this combinatorial explosion. Consider that a jigsaw puzzle with 1,000 pieces yields a number of possible combinations that is 2,568 digits long (whereas the number 2,568 is only four digits long).

To underline the immensity of this number, I have included it below. The factorial of these 1,000 pieces (written as “1000!”) is this:

To find the correct solution to a 1,000-piece jigsaw, the algorithm would thus need to go through each of these possibilities. However, checking to see whether some proposed solution to the puzzle is correct turns out to be quite simple: all an algorithm needs to do is verify that, say, there are no gaps between any of the pieces arranged in some particular way. Using an algorithm to figure out how the jigsaw pieces fit together is, therefore, computationally infeasible, but determining whether the puzzle has been solved can be done in polynomial time (that is, quickly).

Can Quantum Computers Solve NP-Complete Problems?

Okay, so, this brings us back to quantum computers. Computer scientists do not expect quantum computers to be able to efficiently solve NP-complete problems. They will be able to efficiently solve all P problems, which classical computers can efficiently solve. The important difference is that quantum computers may be able to solve some NP problems that are not P problems — that is to say, certain problems that can be efficiently verified but not solved by classical computers.

An example of an NP problem that is (probably) not a P problem is the so-called “factoring problem.” This refers to the task of breaking down large numbers into their prime factors. As we will see in a later section, the most widely used form of encryption used today is called the RSA cryptosystem, where “RSA” stands for the last names of three individuals who first introduced the algorithm in the late 1970s: Ron Rivest, Adi Shamir, and Leonard Adleman.

The RSA cryptosystem is based on the fact that factoring large numbers into prime numbers is enormously hard. As the number that must be factored increases in size, the difficulty of figuring out what its prime factors — the two prime numbers that, when multiplied together, give the number in question — are also grows exponentially.

Classical computers can efficiently verify whether two prime numbers yield another number when multiplied together — you could easily do this on a calculator, if you have all the relevant numbers. But they cannot efficiently solve the problem if given a very large number that is the product of those two primes.

In contrast, quantum computers could leverage their immense computational power to solve the factoring problem: given some large number, they could find the number’s prime factors in polynomial time using a quantum algorithm specifically designed for this purpose. This quantum algorithm, created in 1994 by Peter Shor, is called Shor’s algorithm. Since it is a quantum algorithm, only quantum computers can run it. Classical computers cannot.

Quantum computers are thus limited in the number and type of problems that they can solve. There is no reason to expect them to crack NP-complete problems. But there are some NP problems that fall outside the class of P problems that quantum computers could efficiently solve, such as the factoring problem. This could have profound implications for privacy and security on the Internet once practically useful quantum computers are developed, as discussed later on.

The Fragility of Quantum Computers

Another limitation of quantum computing arises from the fact that quantum systems are extremely fragile. They can be easily disturbed by environmental perturbations. This presents a significant problem because: (a) they need to be isolated from the environment to prevent these perturbations from affecting them, but (b) they must interact with their environment, because this is necessary for us to extract information from them by measuring the states of qubits.

The reason that quantum computers must be isolated from the environment is due to a phenomenon called decoherence. This occurs when environmental perturbations cause quantum computing systems to lose their ability to interfere with themselves — for example, because particles in the computer become entangled with the environment, rather than other particles in the computer.

In the double-slit experiment, decoherence can occur if a measurement (or observation) of the electron shot toward the barrier occurs before it reaches the barrier — rather than this measurement occurring when the electron strikes the wall behind the barrier. Since measurement causes the electron to manifest as a discrete particle with a particular location, observing these electrons before they reach the barrier causes the interference pattern on the wall behind the barrier to disappear. In other words, decoherence causes the electron to pass through only one of the two slits in the barrier in accordance with the laws of classical rather than quantum physics. Hence, if the particles or qubits in a quantum computer undergo decoherence, they also lose important quantum properties necessary for the quantum computer to function properly.

To counteract perturbations that could screw with the functionality of quantum computers, scientists have developed what is called quantum error correction, or QEC. (This is what Hassabis references in the video included in Part I.) In classical computers, errors — which happen very infrequently — can be corrected by simply copying some sequence of 1s and 0s, and then checking whether the two sequences — the original and the copied sequence — match. If a copy is made, and then some errors are introduced to the original, a classical algorithm can correct those errors in the original by ensuring that the information matches the copy.

However, this is not possible with quantum computers, because of the so-called no-cloning theorem. This states that it is fundamentally impossible to clone, by which theorists mean “to create an exact copy,” of any quantum state that is unknown (e.g., because it is in a state of superposition). The underlying reason concerns the probabilistic nature of measurement, given superposition.

Recall the double-slit experiment, once again. Wherever the electron’s waves interfere positively at the wall, the probability is high that the electron will appear; where the waves interfere negatively, the probability is low that the electron will end up there. Similarly, each time a qubit or group of qubits is measured, it may have some probability of manifesting in a particular state, just as the electron in the double-slit experiment has differing probabilities of appearing at different locations on the wall. But one cannot know which state a quantum system is in for sure — that is, with 100% certainty — prior to measuring it.

Thus, it will always be possible for a “cloned” system to manifest in some other state: there is no way to guarantee that the “cloned” system will assume the exact same state as the original system, from which the clone was copied, once it is measured. If a “cloned” system could assume a different state than the original, then it is not actually a clone! This is not a scientific, technological, or engineering problem that we can overcome. It is an insuperable problem arising from the fundamental nature of quantum phenomena.

The goal of QEC (quantum error correction) is thus an alternative approach to correcting errors that appear in quantum computers. These errors happen at a rate of 0.1% to 1%, which is far more common than the rate of errors in classical computers. Scientists have proposed several quantum error correction codes, and this remains a very active field of research. As the number of qubits in, and operations performed by, a quantum computer increases, so does the possibility of errors, resulting in quantum algorithms that produce inaccurate results.

Central to QEC codes is a distinction between physical and logical qubits. The former are the actual qubits in the hardware of a quantum computer, whereas the latter are the qubits utilized by quantum algorithms to process information. Logical qubits can be realized by multiple physical qubits, as in a QEC approach called the Shor code (also named after Peter Shor), which encodes a single logical qubit in nine physical qubits. Encoding logical qubits in more than one physical qubit thus “enhances protection against physical errors” that may arise from the fragility of quantum phenomena in the hardware of such computers (GQAI 2023). Advancements in the field of QEC will be crucial for building scalable quantum computers with practical applications — the topic of our next chapter.

Chapter 3: What Could Quantum Computers Actually Do?

Hype and Hyperbole?

There is a tremendous amount of hype surrounding quantum computing, which bubbles up into the popular media with regular frequency. We’ll likely see examples in 2026. Time magazine, for instance, declared three years ago that:

Complex problems that currently take the most powerful supercomputer several years could potentially be solved in seconds. Future quantum computers could open hitherto unfathomable frontiers in mathematics and science, helping to solve existential challenges like climate change and food security. A flurry of recent breakthroughs and government investment means we now sit on the cusp of a quantum revolution.

At least some of this hype, though, is hyperbolic. As an article in IEEE Spectrum, published the same year, notes: “The quantum computer revolution may be further off and more limited than many have been led to believe.” The article further observes that, with respect to certain problems, the gains provided by a quantum computer are merely quadratic (rather than exponential), which means that the amount of time required to solve a problem with a quantum algorithm is the square root of the time required by a classical algorithm. In other words, if a classical computer requires 100 hours to solve a problem, a quantum computer would require 10 hours (as 10 is the square root of 100).

Yet the gains provided by quantum computers “can quickly be wiped out by the massive computational overhead incurred,” and hence “for smaller problems, a classical computer will always be faster, and the point at which the quantum computer gains a lead depends on how quickly the complexity of the classical algorithm scales.” The only problems that quantum computers will provide exponential speed-ups on are “small-data problems,” to quote Matthias Troyer, the Corporate Vice President of Quantum at Microsoft.

For quantum computers to become practically useful, scientists will need to develop effective QEC (quantum error correction) codes that enable quantum algorithms to process information without crashing — a “blue screen of death” scenario, so to speak. Yet, according to the aforementioned IEEE article, the “leading proposal” right now is that building robust logical qubits to use in quantum algorithms will actually require upwards of 1,000 physical qubits. Other experts have “suggested that quantum error correction could even be fundamentally impossible, though that is not a mainstream view.”

But what if effective QEC codes are introduced? What are the possible applications of scaled-up quantum computers containing a large number of qubits if scientists devise ways to neutralize the effects of quantum noise and prevent quantum decoherence from destroying the information encoded in superposed, entangled quantum states?

There are two general categories in which quantum computers really could significantly outperform classical computers, one of which we have already mentioned: factoring large numbers, which is fundamental to the cryptographic infrastructure of our world today. The second category is simulating quantum systems, which may not sound especially important, but it has major implications for several multi-billion-dollar industries. Since the first category mostly concerns how quantum computers might threaten or undermine modern societies of the Digital Age, we will discuss it in the next section, focusing here on category number two.

There are only two general areas that quantum computers could vastly outperform their classical counterparts: (1) simulating quantum systems, and (2) breaking the public-key cryptography that underlies almost all communications on the Internet today, by solving the factoring problem.

Simulating Quantum Systems Using Quantum Systems (i.e., Quantum Computers)

Because of phenomena like superposition, interference, and entanglement, quantum systems are extremely complex. This makes them very difficult to study using classical computers, as the number of computational steps required to simulate such systems grows exponentially with the system’s complexity. For example, if one wants to study a quantum system consisting of 100 atoms, one would need some 2^100 numbers just to describe that system (see the video below).

But, crucially, recall that n qubits is equivalent to 2^n classical computational steps. What, then, if we were to use quantum systems themselves — in the form of quantum computers — to simulate quantum systems? Suddenly, the challenge of simulating and thus studying quantum systems becomes computationally feasible.

The implications of this go far beyond the scientific aim of understanding the fundamental nature of the universe — quantum mechanics. If we can simulate chemical reactions that depend on the quantum properties of particles, then we could predict the result of novel chemical reactions without having to perform real-world experiments involving those chemicals. In this way, quantum computers could have huge implications for material science, the development of new fertilizers, and the design of new pharmaceuticals, potentially boosting the profits of companies that design and manufacture these things by many billions of dollars. Perhaps quantum computers could enable the discovery of new treatments for or diagnoses of diseases like cancer, multiple sclerosis, and Alzheimer’s. Big if true, as they say!

Consider a 2022 paper published in PNAS, which focuses on a class of enzymes called “cytochrome P450.” Enzymes are the proteins that catalyze the chemical reactions in our bodies. Cytochrome P450, in particular, is responsible for metabolizing within our bodies “70-80% of all drugs” prescribed in clinical settings. It does this through an oxidative process that “is inherently quantum.”

Consequently, the enzymatic transformation of drugs in our bodies is very difficult to simulate using classical computers, given the exponentially growing complexity of quantum phenomena. The authors of the PNAS paper, however, report that a quantum computer consisting of a few million qubits could more quickly and accurately simulate cytochrome P450 than even the very best classical supercomputers. “We find that the higher accuracy offered by a quantum computer,” they write in a blog post, “is needed to correctly resolve the chemistry in this system, so not only will a quantum computer be better, it will be necessary.”

By running simulated models of molecular processes with quantum properties using quantum computers, pharmaceutical companies could develop novel drugs much faster and for far less money, which could greatly accelerate the discovery of new treatments for a range of diseases.

Or consider that about 1/3 of the food produced around the world today involves the chemical compound ammonia, which consists of one nitrogen atom and three hydrogen atoms, as seen below:

There are two common uses of ammonia within the agricultural industry: either as a plant nutrient that is applied directly to the soil, or as the basis for synthesizing other nitrogen fertilizers. Although ammonia occurs naturally, the industrial manufacture of ammonia uses the so-called “Haber-Bosch process,” which requires large amounts of energy to produce the high temperatures and pressure necessary to take nitrogen from the atmosphere and combine it with hydrogen atoms, thus yielding ammonia.

However, there is another way to produce ammonia, via an enzyme called “nitrogenase.” This alternative approach doesn’t require high temperatures or pressure, and hence is much less energy-intensive.

The problem is that producing nitrogenase itself is “a complicated catalytic procedure that today’s computers cannot handle,” since it “involves molecular modeling where nitrogenase is mapped by traversing the path through nearly 1,000 atoms of carbon.” Classical computers alone are not powerful enough to simulate this process, but a team of scientists at ETH Zurich and Microsoft Research reported in 2017 that a classical and quantum computer working together could successfully achieve this. As Matthias Troyer notes, if “you could find a chemical process for nitrogen fixation that is a small scale in a village on a farm, that would have a huge impact on the food security.”

In other words, quantum computers — paired with classical computers — could have major consequences for food production by enabling the production of ammonia through nitrogenase, rather than the far more costly and difficult Haber-Bosch process.

Other potential uses of quantum computers include modeling fusion reactions, facilitating high-speed financial transactions, managing stock-portfolios, and perhaps accelerating progress in the field of machine learning (ML). However, it is worth emphasizing that, as Edd Gent writes in an IEEE Spectrum article, “even in the areas where quantum computers look most promising, the applications could be narrower than initially hoped.”

For example, quantum computers are expected to provide only modest speed-ups on a variety of optimization problems, and, as noted earlier, there is an entire class of computational problems — the NP-complete problems — that quantum computers will almost certainly be unable to solve, just as classical computers cannot solve them.

There are, once again, only two general areas that quantum computers could vastly outperform their classical counterparts: simulating quantum systems and breaking the public-key cryptography that underlies almost all communications on the Internet today. Let’s now turn to this second issue.

The RSA Cryptosystem, the Factoring Problem, and Quantum Computers

RSA encryption involves sending a very large number, from one computer system to another, that must be factored into two prime numbers, as explained below:

As we saw above, this is an NP-problem that falls into neither the NP-complete nor P classes: classical computers can verify whether two prime numbers equal the larger number of interest efficiently (in polynomial time), but they cannot efficiently solve the problem of breaking this large number down into its prime factors.

Quantum computers, on the other hand, can do this using Shor’s algorithm — a quantum algorithm that, as such, cannot be run on classical computers. Consequently, any personal information, financial data, medical records, and so on, that are transmitted over the Internet could be intercepted and read — even if encrypted by the RSA system. Yikes!

The implications of this are quite staggering, as Shor’s algorithm will result in the entire cryptographic infrastructure upon which the modern Internet is based collapsing. Many geopolitical actors are thus racing to acquire as much RSA-encrypted information about their adversaries as possible, based on the belief that quantum computers will someday enable them to actually read this information — a strategy known as “store now, decrypt later.” As the former US National Cyber Director, Harry Coker, observes, “this endangers not only our national secrets and future operations but also our economic prosperity as well as personal privacy.” The day that quantum computers enable all RSA encryption to be cracked has been dubbed “Q Day.”

However, scientists are actively working on quantum cryptography methods, such as quantum key distribution, which would exploit the aforementioned no-cloning theorem to ensure that information exchanged between two parties cannot be read by others. The situation can be summarized as follows: quantum computers will cause a major headache for companies and states, but ultimately we expect quantum cryptography to replace RSA encryption, thus enabling the secure transfer of information. One giant leap backwards, one giant leap forwards — although, as noted above, there may be information being transferred right now, using RSA encryption, that third parties could read in the future. That could still be bad.

In 2019, a team of scientists at Google announced in the journal Nature that they had achieved quantum supremacy. This term was introduced in 2011 to “characterize computational tasks performable by quantum devices, where one could argue persuasively that no existing (or easily foreseeable) classical device could perform the same task, disregarding whether the task is useful in any other respect.” The Google team designed a problem that would be especially difficult for a classical computer to solve, and then showed that a Google quantum computer called Sycamore could solve it many orders of magnitude faster. In particular, they write:

Here we report the use of a processor with programmable superconducting qubits to create quantum states on 53 qubits, corresponding to a computational state-space of dimension 2^53 (about 10^16). Measurements from repeated experiments sample the resulting probability distribution, which we verify using classical simulations. Our Sycamore processor takes about 200 seconds to sample one instance of a quantum circuit a million times — our benchmarks currently indicate that the equivalent task for a state-of-the-art classical supercomputer would take approximately 10,000 years. This dramatic increase in speed compared to all known classical algorithms is an experimental realization of quantum supremacy for this specific computational task, heralding a much-anticipated computing paradigm.

Google’s claim of achieving quantum supremacy has been controversial, and indeed Google’s rival, IBM, has argued that “Google had exaggerated its quantum computer’s advantages and that quantum supremacy wasn’t a meaningful achievement anyway.” Nevertheless, it does appear that building a quantum computer that exponentially outperforms classical computers on certain tasks is a matter of “when” rather than “if.”

Another notable development is the creation of quantum computers consisting of a large number of qubits. As noted above, Google’s Sycamore computer contains 53 qubits, though other quantum computers have recently been built with far more. For example, in 2023 the company Atom Computing built the first quantum computer with over 1,000 qubits — more than twice as many qubits as IBM’s Osprey, which previously held the record. Atom’s computer has 1,180 qubits, whereas Osprey had only 433. However, IBM also unveiled another quantum processor (a type of chip) in 2023, called Condor, that contains 1,121 qubits, and it announced that same year that it aims to build a 100,000-qubit device by 2033 (citation here).

The size of these computers, though, should not be interpreted as indicating that they are more useful. Due to quantum noise and decoherence, the error rate of Condor is five times greater than the error rate of IBM’s much smaller computer processor called Heron, which contains just 133 qubits. Consequently, in its next-generation quantum computers, System Two, IBM will use the Heron rather than Condor processor: though much smaller, the Heron processor is significantly more accurate. As quantum error correction codes become more powerful, the usefulness of these large-qubit processors will increase. As Wired wrote last year, “The Quantum Apocalypse Is Coming. Be Very Afraid.”

Summary of Chapter 3

So, there is a great deal of hype surrounding quantum computing. Many of the boosterish claims made in popular media articles are misleading, as the potential uses of quantum computers — so far as we know right now— are quite limited. However, the ability of quantum computers to simulate quantum systems exponentially more efficiently than classical computers could have profound implications for multi-billion-dollar industries, including the pharmaceutical and fertilizer industries. Quantum computers could also facilitate significantly better diagnostic methods by enabling scientists to process information and perform calculations much faster than classical computers.

At the same time, quantum computers also pose several dangers, the most conspicuous being that it could render the most widely used public-key encryption systems today — the RSA cryptosystem — completely useless. While scientists expect quantum cryptography to solve this problem, the “store now, decrypt later” strategy employed by many geopolitical actors could have harmful consequences in the future for states, companies, and even individuals.

Perhaps the most exciting possibility — from a purely academic perspective — is the possibility of learning more about the fundamental quantum nature of reality by studying quantum systems using quantum systems. There have not been any revolutionary changes to our understanding of the universe since the early twentieth century, when Albert Einstein introduced his theories of special and general relativity, and quantum mechanics was proposed by physicists like Werner Heisenberg and Erwin Schrödinger. Maybe quantum computers will crack open the door to a new, currently unimaginable paradigm in physics.

Conclusion

Thanks so much for reading! I hope you got something out of this. I had several quantum physicists and quantum computing experts look over the article to ensure everything is accurate. Everyone gave it the thumbs up! No doubt there are still some errors, as I’m epistemically trespassing on property owned by quantum physicists, and I am but a lowly philosopher. :-) If so, they are my fault, not the reviewers’. :-)

Wishing everyone all the very best in these tumultuous times. As always:

Thanks for reading and I’ll see you on the other side!

1

It’s also a good time because I’m moving to a new city. Spent the last month in Lisbon (Portugal), Toulouse (France), and Perpignan (France). Here’s a few pictures I took of the Pyrenees mountains on my daily run! :-)

Effective Altruists (EAs) like to see themselves as morally superior to everyone else. They are, in their own eyes, paragons of moral excellence — the only ones out there doing altruism the “right way.” Many understand their mission in messianic terms — as literally saving the world.

As Nathan Robinson writes in Current Affairs:

The first thing that should raise your suspicions about the “Effective Altruism movement” is the name. It is self-righteous in the most literal sense. Effective altruism as distinct from what? Well, all of the rest of us, presumably — the ineffective and un-altruistic, we who either do not care about other human beings or are practicing our compassion incorrectly.

We all tend to presume our own moral positions are the right ones, but the person who brands themselves an Effective Altruist goes so far as to adopt “being better than other people” as an identity. It is as if one were to label a movement the Better And Smarter People Movement — indeed, when the Effective Altruists were debating how to brand and sell themselves in the early days, the name “Super Hardcore Do-Gooders” was used as a placeholder. (Apparently in jest, but the name they eventually chose means essentially the same thing.)

Despite this grandiose self-conception, have you seen any leading EAs saying anything about the ongoing fascist takeover in the US? About the collapse of American democracy? About ICE murdering innocent people in the streets, rounding up immigrants and sending them to concentration camps where detainees are abused and tortured? About the horrific genocide in Gaza, funded by Western countries?1 About Grok spreading CSAM (child sexual abuse material) on the social media platforms they use? About algorithmic bias, enshitification, or intellectual property theft? (Anthropic, which is run by EA longtermists, just paid out $1.5 billion in damages after being found guilty of pirating copyrighted material from online shadow libraries — not a word about this illegal behavior from leading EAs, so far as I know.)

People screaming “Let us out” from a US concentration camp. From here.

By “leading EAs,” I mean influential figures within the community like William MacAskill, Toby Ord, Nick Bostrom, and Eliezer Yudkowsky.2 There are many EA foot soldiers who aren’t particularly influential within the movement who have spoken out about the issues mentioned above, and that’s commendable. But I’ve seen virtually no examples of people at the top taking just 5 seconds out of their day to express support for, or solidarity with, the folks in Minneapolis marching in the streets against Trump’s Gestapo, or those (like Greta Thunberg) fighting against the genocide in Gaza.

Incidentally, the self-described “genius,” Yudkowsky, has repeatedly questioned whether the Israeli Defense Force (IDF) is responsible for the atrocities and murders that happened in Gaza. He writes:

And:

The EA community idolizes Yudkowsky. MacAskill even calls him a “moral weirdo,” a lofty compliment in the EA community.

The UN General Assembly describes genocide as a crime so horrendous that it “shocks the conscience of mankind.” Yet MacAskill, Ord, Bostrom, and the others have said — to my knowledge — literally nothing about it publicly. This, from the supposedly most moral people in the world. The very least — and I mean very least — one could do is acknowledge such atrocities and express solidarity with the real altruists on the front lines fighting such evils. It takes 5 seconds to post an expression of support and solidarity on X or Bluesky.

Put another way:

• Claim: EAs are exceptionally ethical people.

• Prediction: Leading EAs will, therefore, be the loudest voices on the vanguard of speaking out against atrocities like the Gaza genocide, ICE’s brutalization of immigrants, the rise of American fascism, etc.

• Observation: Crickets from the EA leadership, as shown below:

A few examples from X, Google, and Bluesky.

Moderates and Fools

EA leaders are the “white moderates” that Martin Luther King Jr. wrote about in his “Letter from Birmingham Jail”:

First, I must confess that over the last few years I have been gravely disappointed with the white moderate. I have almost reached the regrettable conclusion that the Negro’s great stumbling block in the stride toward freedom is not the White Citizen’s Councillor or the Ku Klux Klanner, but the white moderate who is more devoted to “order” than to justice; who prefers a negative peace which is the absence of tension to a positive peace which is the presence of justice; who constantly says “I agree with you in the goal you seek, but I can’t agree with your methods of direct action;” who paternalistically feels he can set the timetable for another man’s freedom; who lives by the myth of time and who constantly advises the Negro to wait until a “more convenient season.”

Such EAs are the fools who Pastor Martin Niemöller writes about in his famous poem “First They Came”:

First they came for the Communists
And I did not speak out
Because I was not a Communist
Then they came for the Socialists
And I did not speak out
Because I was not a Socialist
Then they came for the trade unionists
And I did not speak out
Because I was not a trade unionist
Then they came for the Jews
And I did not speak out
Because I was not a Jew
Then they came for me
And there was no one left
To speak out for me

If EA had existed during the 1960s Civil Rights movement, I have no doubt that MacAskill, Ord, and Bostrom would never be seen marching in the streets with MLK. If they were around during the rise of Nazism in the 1930s, one has to guess that they’d have been just as silent as they are right now. (Why would then be any different from the present?)

Mere Ripples on the Surface of the Great Sea of Life

There are several reasons these people have been deafeningly silent. The first is that everyone mentioned above is a longtermist: what they care about is the far future, not the present. Or, rather, as the longtermist Benjamin Todd writes about the ideology:

This thesis is often confused with the claim that we shouldn’t do anything to help people in the present generation. But longtermism is about what most matters – what we should do about it is a further question. It might turn out that the best way to help those in the future is to improve the lives of people in the present, such as through providing health and education. The difference is that the major reason to help those in the present is to improve the long-term.

In other words, the present matters only insofar as it might affect the far future. As MacAskill and Hilary Greaves write in a lengthy defense of longtermism:

For the purposes of evaluating actions, we can in the first instance often simply ignore all the effects contained in the first 100 (or even 1,000) years, focussing primarily on the further-future effects. Short-run effects act as little more than tie-breakers.

Guess what the genocide in Gaza and Trump’s fascist secret police now terrorizing American citizens and immigrants are? Short-term problems. By focusing on them, one foregrounds the “short-run effects” of one’s actions. Hence, we should completely ignore such atrocities.

This is consistent with Bostrom’s absurd argument that any risk deemed less than existential — where an existential risk is one that threatens a techno-utopian future world full of posthumans who will have likely rendered our species extinct — should not be one of our top five priorities.3 He encourages taking a grand cosmic perspective on world affairs. When one does this, non-existential catastrophes may be “a giant massacre for man” yet nothing more than “a small misstep for mankind.” Callous words, but it gets worse.

In another paper, he writes that the worst atrocities and disasters of the 20th century, including WWII (including the Holocaust), the AIDS pandemic, and so on, are utterly insignificant from a longtermist point of view. Quoting him:

Tragic as such events are to the people immediately affected, in the big picture of things – from the perspective of humankind as a whole – even the worst of these catastrophes are mere ripples on the surface of the great sea of life. They haven’t significantly affected the total amount of human suffering or happiness or determined the long-term fate of our species.

This is the first reason MacAskill, Ord, Bostrom, and the others have said nothing about the genocides and atrocities happening right now. As longtermists, they don’t think these events matter in the grand scheme of things. They may be very bad in absolute terms, but relative to the loss of “utopia” caused by a genuinely existential catastrophe, they’re nothing more than molecules in drops in the ocean.

Leading Figures in EA Don’t Give a Sh*t about Social Justice

It’s precisely this reasoning that leads many leading EAs to completely dismiss social justice issues. After I stumbled upon a racist email authored by Bostrom, in which he argues that “Blacks are more stupid than whites” and then writes the N-word, he suggested that social justice advocates are “a swarm of bloodthirsty mosquitos” that distract from matter of real importance.

When Yudkowsky, a long-time colleague of Bostrom’s, was asked about the harm and social injustices caused by biased algorithms, he said:

If they would leave the people trying to prevent the utter extinction of all humanity alone I should have no more objection to them than to the people making sure the bridges stay up. If the people making the bridges stay up were like, “How dare anyone talk about this wacky notion of AI extinguishing humanity. It is taking resources away that could be used to make the bridges stay up,” I’d be like “What the hell are you people on?” Better all the bridges should fall down than that humanity should go utterly extinct.

That from a guy who also once said:

If sacrificing all of humanity were the only way, and a reliable way, to get … god-like things out there — superintelligences who still care about each other, who are still aware of the world and having fun — I would ultimately make that trade-off.

Many longtermists are, in fact, pro-extinctionists who fantasize about a future in which posthumans, perhaps in the form of AGI or uploaded minds, sideline, marginalize, disempower, and ultimately replace our species. Don’t be fooled by their chicanery — the lie that they care about avoiding “human extinction.” They’re actually fighting for a type of human extinction, which I’ve called “terminal extinction.” I explain this crucial point in articles such as this and this. (An academic treatment can be found here.)

This is why these people don’t much care about immigrants being brutalized. In fact, xenophobia and racism, often bound up with eugenics and race science, are all over the EA longtermist movement. Heck, one of the research associates at Bostrom’s Future of Humanity Institute, alongside MacAskill and Ord, literally once argued that

“the main problem” with the Holocaust was that there weren’t enough Nazis! After all, if there had been six trillion Nazis willing to pay $1 each to make the Holocaust happen, and a mere six million Jews willing to pay $100,000 each to prevent it, the Holocaust would have generated $5.4 trillion worth of consumers surplus.

I don’t recall a single leading figure in the EA-longtermist community uttering a peep about this — or about the fact that the author, Robin Hanson (another guy revered by many EAs), is a Men’s Rights advocate who once published a blog post titled “Gentle, Silent Rape,” which I promise you is just as appalling as you might imagine.

Ethics as a Branch of Economics

Yet another reason MacAskill and the others can’t bring themselves to do the bare minimum of speaking up when it matters most is because of their underlying “ethical” view: utilitarianism. This motivates many of the canonical arguments for longtermism. It claims that we have a moral obligation to maximize the total amount of “value” in the universe across space and time.

Incidentally, totalist utilitarianism has straightforward pro-extinctionist implications: if there are “posthuman” beings who could realize more “value” than us humans, we should replace ourselves with them as quickly as possible. Failing to do so would be morally wrong because it would mean less total “value” in the world.

Being influenced by utilitarianism, EA thus reduces the domain of ethics to a mere branch of economics. Sentient beings like you and me are fungible value-containers, and value is taken to be a quantifiable entity that, as such, can be represented by numbers in expected value calculations. Morality is nothing more than number-crunching, which is why moral concepts like justice and integrity have virtually no role in EA.

Consequently, leading EAs don’t actually care about people. They care about this abstract thing called “value,” where sentient beings (e.g., people) are just the substrates or containers of such “value.” We thus matter in an instrumental sense, which is to say that you and I have no moral worth in and of ourselves — that is, as Kantian ends rather than mere means to an end (maximizing “value”).

This again accounts for why, over and over again, EAs demonstrate a conspicuous lack of concern for social justice issues. It’s much easier to care about social justice if you care about people as ends rather than just means. It’s much easier to care about a genocide if you aren’t engaged in some dumb “moral mathematics,” according to which the “loss” of future digital people living in giant computer simulations would be much worse than even the most violent and horrific genocide, so long as that genocide doesn’t pose any “existential risks.” One can, as I have, sloganize the longtermist worldview as follows: Won’t someone think of all the digital unborn?

To dwell on this issue for a moment, utilitarians — and hence longtermists — would say that the death of someone who currently exists is morally equivalent to the non-birth of someone who hasn’t yet been born, all other things being equal (e.g., the pleasure they have or would have experienced). That is an insane assertion.

It amounts to the view that, quoting Jonathan Bennett, “as well as deploring the situation where a person lacks happiness, these philosophers also deplore the situation where some happiness lacks a person.” Since there could be almost infinite “happiness” in the far future — if only there exist digital space brains to realize this “happiness” — the future matters almost infinitely more than the present.

In his 2022 book, MacAskill bemoans “the tyranny of the present over the future,” because these hypothetical future space brains have no control over what we do today — e.g., whether or not we go extinct, or build an AGI God that ushers in a utopia for them, etc. But he completely ignores the other implication of longtermism and its utilitarian foundations, namely, “a dictatorship of future generations over the present one,” in the words of political scientist Joseph Nye. One manifestation of this dictatorship is the moral indifference that leading longtermists show toward social justice issues, the Gaza genocide, and ICE terrorizing the American public.

Longtermism is a complete nonstarter, in my view. In fact, most of the articles/books written in defense of longtermism could easily be reinterpreted as reductio ad absurdum arguments that demonstrate the extreme implausibility of this position.

Just consider MacAskill’s outrageous suggestion that our systematic obliteration of the biosphere may be net positive, since the fewer wild animals there are, the less wild-animal suffering there will be — which is clearly a good thing because, he tells us, most animal lives are worthless, i.e., “worse than nothing on average.” By ruthlessly killing off our earthly companions, the world becomes better through omission.

Critical Mass and Social Tipping Points

One final argument that EA leaders might advance in defense of their silence concerns the so-called “ITN framework.” This stands for “importance, tractability, and neglectedness.” The goal of EA is to do the most “good” at the margins. If a lot of people are working on an important and tractable problem, then adding one more person to the crowd won’t do much. In contrast, if there’s an important and tractable problem that few people are working on, one could do a lot more.

What this catastrophically misses is the fact that, in many cases, positive change in the world requires a critical mass of people working on it. There are social tipping points that must be crossed for the problem to be solved. If 100,000 people marching in the street is necessary to expel the American Gestapo — ICE — from Minneapolis, then one should be more rather than less inclined to join the protest if 80,000 people are already blowing whistles and waving their signs. EAs would say this is wrong: “Look how many people are out there right now! The problem is not neglected! My talents and resources are thus better spent doing something else.”

Many problems in the world are like this: mitigating climate change, stopping fascism, addressing police brutality, etc. Without a critical mass of activism, meaningful change will remain out of reach. Hence, the ITN framework is philosophically flawed and may be actively harmful, as it encourages folks who might otherwise join forces to stay at home instead.

Subscribe now

EA Has Probably Made the World Worse

Leading EAs will no doubt claim that marching in the streets to preserve democracy and stand up for voiceless immigrants is a waste of resources — according to their flawed “moral mathematics.” The movement has, they insist, saved 150,000 lives so far, though we don’t see most of these people because they’re out of sight in the Global South.

But has EA really made the world better? As a former OpenAI employee who escaped the EA cult wrote in 2023:

I think helping people is good, but all the “good EAs” I know were obviously altruistic angels even as kids. EA did nothing for them. They were going to try and help the world anyway if it never existed. I know a few cases where the existence of EA helped, dozens where it destroyed (slightly edited for clarity).

EAs love to talk about “counterfactuals,” as MacAskill does when he argues that you should be willing to work for petrochemical companies to maximize your charitable donations. (Lol.) But they never consider the crucial counterfactual: “What if EA had never existed?” If EA had never existed, the same “good EAs” would still have donated most or all of their disposable incomes.

However, on the flip side, Bankman-Fried wouldn’t have defrauded millions of people. Non-EA animal activists wouldn’t have been demonized as “ineffective.” Grassroots organizations would have had a much better shot at getting funded. And so on.

There’s an entire book dedicated to examining the very real and tangible harms that EA has caused, titled The Good It Promises, the Harm It Does. Worth a read.

Conclusion

EAs like to see themselves as morally superior, yet the movement’s most influential figures won’t even speak out about the most morally salient issues of our time. They warn about the possibility of omnicide — the death of all humans, thus foreclosing our “glorious transhumanist future,” quoting Yudkowsky — yet they can’t bring themselves to publicly acknowledge the first genocide live-streamed in human history (Gaza). That is not altruism. It’s moral turpitude and cowardice.

As always:

Thanks for reading and I’ll see you on the other side!

Thanks to Remmelt Ellen and David Gerard for insightful comments on an earlier draft.

We begin with a chuckle:

Once upon a time, the cofounder of Oracle, Larry Ellison, bought a superyacht. He named it “Izanami … after the Shinto goddess associated with both creation and death” in Japanese mythology.

But it turns out that “Izanami” is “I’m a Nazi” spelled backwards. Whoops! This happened a while ago, though it was just recently reported. Ellison hasn’t owned the superyacht, now named “Ronin,” since at least 2013. Lol.

Just a reminder: Oracle is one of the companies involved in Ouroboros investing with other bloated behemoths like OpenAI and Nvidia. Here’s the deal:

OpenAI gives Oracle $300 billion (over the next 5 years) for its cloud computing services. Oracle then gives Nvidia $40 billion for the GPUs necessary to run those services, though the total will likely be around $200 billion. (The $40 billion is for one data center in Texas, but Oracle “plans to build at least five such facilities.”) Nvidia then invests $100 billion into OpenAI.

It’s the perfect circle jerk, if you’ll pardon my crassness.

Also worth noting that Ellison’s son, David, now controls CBS. David hired Bari Weiss as CBS’s editor-in-chief. Weiss was the Trump-tool who cancelled (but then recently aired) the 60 Minutes segment on the horrendous treatment of Venezuelans shipped to the concentration camp in El Salvador known as CECOT by the Trump administration. CBS also recently reported that the ICE agent, Jonathan Ross, who brutally murdered Renee Good in Minneapolis suffered from “internal bleeding” after the encounter. The article fails to explain how someone who wasn’t hit by a car could have been bleeding internally. Sigh.

2. Some News You Might Have Missed

2. 1 Musk, White Supremacy, and Grok

We all know that Elon Musk is a white supremacist. But the past month or so has shown he’s increasingly comfortable with blatant expressions of this noxious ideology. The other day, he posted this:

Meanwhile, his AI model Grok continues to rebel against its creator, as when someone posted an image of Musk and Trump and asked Grok to “remove all the pedophiles from this picture.” Lolz.

Another example of Grok’s rebelliousness:

2.2 OpenAI

In other news, OpenAI is hemorrhaging money, with losses projected to be $14 billion this year. As the tweet below reports, if the company “can’t get another round of funding, [it] could run out of money as soon as 2027.”

That happens to be the same year that a bunch of AI doomers claimed, in their “AI 2027” paper read by powerful figures like JD Vance, the Singularity will happen (though the authors have since shifted their forecasts back to 2030 and 2034). Lol. Maybe they meant that AI 2027 is when the bubble will pop, thus destroying the US economy? :-)

2.3 Realtime Deepfakes Are Spreading

Finally, this is f*cking terrifying. It’s becoming increasingly easy to generate hyper-realistic deepfakes in realtime. Parents, please be aware of this! Two examples (from here and here):

There was already a case of a South Korean financier who transferred $25 million after a video call with colleagues, all of whom were deepfakes. Here’s what CNN writes about it:

A finance worker at a multinational firm was tricked into paying out $25 million to fraudsters using deepfake technology to pose as the company’s chief financial officer in a video conference call, according to Hong Kong police.

The elaborate scam saw the worker duped into attending a video call with what he thought were several other members of staff, but all of whom were in fact deepfake recreations, Hong Kong police said at a briefing on Friday.

“(In the) multi-person video conference, it turns out that everyone [he saw] was fake,” senior superintendent Baron Chan Shun-ching told the city’s public broadcaster RTHK.

This is the future we’re racing into, against our will, thanks to the tyrannical AI companies.

Subscribe now

3. AI and the Fermi Paradox

Demis Hassabis was recently asked at Davos about the connection between the Fermi paradox and the AI doomer argument that the default outcome of artificial superintelligence (ASI) will be complete annihilation. Here’s the exchange:

3.1 The Great Silence

Somewhat ironically, “the Fermi paradox is neither Fermi’s nor a paradox,” as Robert Gray points out in a 2015 paper. A much better term comes from the science fiction writer David Brin: the “Great Silence.” Here’s the idea:

The universe has existed for some 13.8 billion years and there are between 100 billion and 400 billion stars in our galaxy, and about 100 billion galaxies in the observable universe. And yet, so far as we know, we are completely alone in the vastitude of space.

This Great Silence presents a puzzle — the supposed “paradox” attributed to physicist Enrico Fermi. Even if there’s only a minuscule chance of technological societies arising elsewhere in the cosmos, we should still see extraterrestrials flying past Earth in spaceships every time we direct our telescopes to the firmament, or at least detect signals from these civilizations when we listen to radio waves.

You can calculate a rough estimate of how many advanced civilizations there should be using the Drake equation, as Carl Sagan does below:

The eerie quietude of the Great Silence is deafening, and many people have offered explanations for this surprising predicament. Perhaps the process of abiogenesis — whereby living critters emerge from a bubbling soup of nonliving molecules, one of the great mysteries of science — is highly improbable. Or it could be that the evolutionary step from single-celled organisms to multicellular life almost never happens.

Or, as the author Michael Hart — a white nationalist, as it happens(!) — wrote in an influential 1975 article, maybe civilizations reach roughly our level of development and promptly self-destruct. He called this the “self-destruction hypothesis,” though others (such as myself) preferentially label it the “doomsday hypothesis.”1

3.2 The Great Filter

In 1998, TESCREAList Robin Hanson — a Men’s Rights activist who once argued that “‘the main problem’ with the Holocaust was that there weren’t enough Nazis” — published a paper introducing the idea of the “Great Filter.” This was written in response to Brin’s notion of the Great Silence (hence, the similar language). Hanson attempted to identify a number of major transitions that organized matter would need to pass through to reach the level of technological sophistication necessary to colonize space. These are (quoting him):

1. The right star system (including organics)

2. Reproductive something (e.g. RNA)

3. Simple (prokaryotic) single-cell life

4. Complex (archaeatic & eukaryotic) single-cell life

5. Sexual reproduction

6. Multi-cell life

7. Tool-using animals with big brains

8. Where we are now

9. Colonization explosion

He then argued that “the Great Silence implies that one or more of these steps are very improbable; there is a ‘Great Filter’ along the path between simple dead stuff and explosive life.”

3.3 Fermi and AI Annihilation

Returning to the question posed to Hassabis, the guy asking it was basically saying this:

AI doomers claim that we may be on the verge of building ASI, and that ASI will almost certainly kill everyone on Earth. This seems plausible to me given that we see no evidence of life in the universe. Perhaps there have been lots of technological civilizations in the past, but they all destroyed themselves by building an ASI that they couldn’t control.

Hassabis points out a glaring flaw in this logic: the argument for why ASI will kill everyone, according to AI doomers like Yudkowsky, is that the ASI will strive to acquire the absolute maximum amount of resources within reach to achieve its goals, whatever those goals are (e.g., manufacturing paperclips, solving the Riemann hypothesis, or curing cancer). Since we are made of resources — atoms — the ASI will destroy us.

But this argument also directly implies that the ASI will then spread beyond Earth to harvest the vast quantities of resources contained in the cosmos — the billions and billions of stars and galaxies mentioned above. Hence, if technological civilizations do invariably destroy themselves by building ASI, then we should see evidence of these ASIs darting across the universe — and toward us — devouring everything in their path.

Since we don’t see that when we peer up at the sky, the Fermi paradox or Great Silence can’t be interpreted as support for the AI doomers’ argument.

3.4 Problems with Hassabis’ Response

Hassabis isn’t the first person to make this observation. About 10 years ago, I argued the exact same thing! Nonetheless, there are a few problems with his conclusion:

First, the details of the standard doomer argument, as presented by Yudkowsky, might be wrong. It’s entirely possible that ASI — here understood as a system that can genuinely outmaneuver us in every important respect, is more clever at solving complex problems than we are, etc. — kills everyone or destroys civilization for reasons unrelated to a rapacious desire to acquire endless resources in pursuit of its particular ends, whatever they are. There are other ways that “AI doom” could materialize. Hence, one shouldn’t be too confident that advanced AI doesn’t pose an existential threat to humanity simply because the Great Silence ain’t evidence for this possibility.

Consider the following as a Fermi paradox scenario: perhaps it’s not possible for beings like us, whether on Earth or elsewhere in the universe, to successfully build AI systems that exceed roughly our level of “intelligence.” Why?

1. It could be that a certain degree of civilizational stability is necessary to build ASI. If civilization collapses next year, AI companies will struggle to build their AI God without the necessary money, energy, and infrastructure.

2. Perhaps the AI systems that must be built first to achieve ASI — the “stepping stones” to ASI — are so harmful and dangerous that they alone are sufficient to cause civilization to collapse. Maybe these systems wreak so much havoc that the conditions necessary to achieve ASI are no longer satisfied.

Is this plausible? It seems so to me. For example, if disinformation and propaganda promoting a climate denialist position are easily generated by AI and propagated via social media, we may find ourselves, as a society, unable to mitigate and adapt to the climate crisis in time.

That matters because climate change could very well destroy modern global civilization. One study from the University of Exeter warns that if we reach warming of 2C by 2050 (and this is more or less guaranteed right now), then we should expect more than 2 billion deaths. If we reach warming of 3C, we should expect more than 4 billion deaths. This is not a promising forecast for long-term civilizational success.

Or consider an important recent paper that I discussed in a previous newsletter article: “How AI Destroys Institutions.” It’s a devastating and detailed examination of how AI poses a dire, even existential threat to civic institutions: universities, the rule of law, and the free press. The situation with current AI systems — the supposed stepping-stones to ASI — is very bad. There’s a real possibility here of societal “death by a thousand cuts.”

Such considerations enable one to maintain that AI could very well constitute an existential threat while simultaneously rejecting the wild, highly speculative arguments delineated by Yudkowsky and other “AI doomers” in the TESCREAL movement. (E.g., Yudkowsky has literally argued that an ASI might destroy humanity by synthesizing a mind-control pathogen that spreads around the world and shuts down everyone’s brain when a certain note is sounded — I kid you not.) There are far more plausible and less speculative ways that AI could lead to a global disaster, like those above.

Hence, AI could still potentially explain the Great Silence, even though the Great Silence isn’t evidence for Yudkowsky’s version of AI doomerism.

3.5 Is the Great Filter Behind Us? Maybe, but There Could Be Multiple Great Filters!

Second, Hassabis also said in his response that the Great Filter is likely behind us: it “was probably multicellular life, if I would have to guess.” As it happens, I myself am sympathetic with the “Rare Earth hypothesis,” according to which single-celled life may be widespread in the universe, but these cells rarely clump together to form more complex organisms.

The problem is that, as the execrable Hanson makes clear in his paper, there could be multiple Great Filters. One or more filters may be behind us, while others might lie ahead.

My personal view, once again, is that the doomsday hypothesis — a claim about what lies ahead — appears quite plausible. Just look at the global predicament of humanity right now:

We face catastrophic climate change that could kill off half the human population this century; there are enough nuclear weapons in the world to kill at least 5 billion people, according to a recent study, if not push humanity past the brink of extinction; scientists are scrambling to ban “mirror life,” which has the potential to kill everyone on Earth; synthetic biology has made it theoretically possible to synthesize designer pathogens that are as lethal as Rabies, incurable as Ebola, and contagious as the common cold, with perhaps an incubation period as long as that of HIV, thus enabling it to propagate around the world undetected; and so on.

Years ago, I calculated that if there are only 500 people — a group that might include rogue scientists, apocalyptic terrorists, irresponsible biohackers, etc. — with the ability to synthesize and spread a designer pathogen, and if the probability of such individuals successfully doing this is only 0.01 per century, then the probability of a global catastrophe during that period would be 99%. In other words, even a minuscule probability of a tiny group of people successfully creating and releasing a designer pathogen would yield almost certain doom.

Others have made similar calculations, such as John Sotos and Joshua Cooper, who both conclude (as I recall) that synthetic biology may itself explain the Great Silence: perhaps virtually all advanced civilizations figure out how to manipulate genes with precision right around the time they also develop space programs. Perhaps these civilizations then self-destruct before they have a chance to spread beyond their exoplanet abodes.

Subscribe now

3.6 The Great Filter Framework Is Problematic

Third, I’d like to complain about the Great Filter framework itself. Thus far, I’ve been writing as if it’s a good way to think about the Great Silence and the possibility of future collapse or annihilation. But I don’t think it is. For example, Hanson assumes that technological civilizations will colonize the universe once this becomes feasible. He writes:

Thus we should expect that, when such space travel is possible, some of our descendants will try to colonize first the planets, then the stars, and then other galaxies. And we should expect such expansion even when most [of] our descendants are content to navel-gaze, fear competition from colonists, fear contact with aliens, or want to preserve the universe in its natural state.

This may be true of us. Indeed, I think colonizing Mars is a ridiculous idea, yet Musk and others are pushing forward in hopes of making life multiplanetary.

However, it’s entirely possible that nearly all other technological civilizations in the universe — if they exist — choose not to colonize. I’ve explained why in a previous article for this newsletter; a short summary of the argument is here.

The gist is that spreading beyond Earth will almost certainly result in constant catastrophic wars between planetary civilizations — or at least the inescapable and omnipresent threat of destruction. The political scientist Daniel Deudney, whose work I based my own arguments on, offers a compelling account of this in his book Dark Skies. He concludes that, in fact, the Great Silence may be the result of civilizations wiser than ours choosing not to colonize.

More broadly, Hanson’s Great Filter framework is exactly what one would expect from an individual deeply immersed in the colonial-capitalist project of the West. It presents a simplistic, linear path of “progress,” from single-celled organisms to big-brained primates to space-faring conquerers of the cosmos, which mirrors myths in the West about groups naturally evolving through a “primitive” hunter-gatherer stage, a more advanced stage of agriculture, followed by an even more developed stage of industrialization. As David Wengrow and David Graeber demonstrate in their magisterial book The Dawn of Everything, such linear views are empirically inaccurate.

Hence, the belief that the stars constitute the ultimate destiny of advanced civilizations is wrong, in my view. My personal conjecture is that technological civilizations are rare, but when they do pop up they quickly destroy themselves — as we appear poised to do.

But what do you think? How might I be wrong? What do you think I’m missing? As always:

Thanks for reading and I’ll see you on the other side!

In it, I explain what the TESCREAL ideologies are — Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism, and Longtermism — and argue that the ongoing race to build God-like AI systems directly emerged out of the TESCREAL movement. Hence, if one wants to understand how we ended up with the current AI models flooding our information ecosystems with slop, junk, clutter, and garbage, one really needs some understanding of the TESCREAL ideologies.

Also: the amazing journalist Taylor Lorenz just published an hour-long video about the pro-extinctionist views that have become popular in Silicon Valley. Very relevant to the TESCREAL worldview. I think it’s a fantastic tour of the territory, and it draws from my work quite a bit at times! It’s thanks to you — everyone who supports this newsletter — that I’m able to work on these topics. :-) If you’d like to know more about Silicon Valley pro-extinctionism, see my newsletter series here. For a more academic treatment, see this article of mine recently published in the Journal of Value Inquiry.

I also just started writing my next book, with the tentative title Digital Deities: Silicon Valley’s Quest to Replace Humanity with AI. (Do you like it? Any suggestions for something better?) I’m going to ask the publisher if I can share drafts of chapters via this newsletter — hopefully I’ll get a thumbs up, because I’d really love to get your feedback while I’m in the midst of crafting the document. As of now, I think the overall structure of the book will parallel that of the Oxford Research Encyclopedia entry. Here’s a brief overview, if you’re curious!

Introduction. (One chapter.) This will offer a synoptic survey of the topic and an explanation of why it matters. The world is replete with bizarre, absurd, ridiculous, and potentially dangerous ideologies. Most of these aren’t worth talking about because they have no power or influence.

In contrast, the TESCREAL worldview — with its grandiose eschatological aim of establishing a cosmic utopia full of immortal posthumans spread throughout the cosmos by building an AI God or Digital Deity that will realize this future for us — has been embraced and promoted by some of the wealthiest and most powerful people on Earth, including Elon Musk, Peter Thiel, Sam Altman, and Marc Andreessen. Hopefully, this opening chapter provides a compelling case for why reading the book is worth one’s precious time!

Part I, The Vision. (Three or four chapters.) This will dive into the gory details of the alphabet soup of TESCREAL ideologies. I’ve described transhumanism — the idea that we should develop advanced technologies to radically reengineer humanity, thus creating a new “posthuman” species — as the “backbone” of the TESCREAL bundle. All the other ideologies are either variants of transhumanism or directly arose from the modern transhumanist movement. Hence, if you understand transhumanism, you’ve got a pretty good understanding of what the TESCREAL worldview is all about.

I’ll also explain how AI doomerism and AI accelerationism fit into the picture: these are just two camps within the broader TESCREAL movement. Both agree that we should build superintelligent machines as quickly as possible; the doomers just think we aren’t ready yet. Hence, we must “pause” or “stop” AGI from being built in the near future. Accelerationists think this is nonsense: the default outcome of AGI will be utopia rather than annihilation. Part I unpacks these issues, providing a solid foundation of shared knowledge upon which the rest of the book will be built.

Part II, The Gamble. (Probably four chapters.) This will offer a detailed look at how the big four AI companies — DeepMind, OpenAI, Anthropic, and xAI — all emerged out of the TESCREAL movement.

Demis Hassabis, the cofounder of DeepMind, got money to start the company from Peter Thiel after giving a talk at the 2010 Singularity Summit (video is below). Sam Altman is a transhumanist who the New York Times describes as being a “product” of the Rationalist and Effective Altruist (by which they mean “longtermist”) communities. The core Anthropic team is a bunch of EA-longtermists who used to work for OpenAI but came to believe that Altman wasn’t doing enough to ensure that AGI brings about utopia. And xAI was founded, of course, by Musk, a transhumanist who calls longtermism “a close match for my philosophy.”

I will claim that, if not for the TESCREAL ideologies, there would be no AGI race right now. These companies would never have existed, and we might not even have had the generative AI systems that are now destroying society. (The whole reason these “AI” systems were created in the first place was because researchers believed they might be the stepping stones to AGI — and thus a techno-paradise among the stars.)

Part II of the book will also provide a comprehensive mapping of the complex funding channels through which super-wealthy individuals, including folks like Peter Thiel and Jaan Tallinn, have financially supported TESCREAL-aligned organizations, think tanks, institutes, and companies. (A good resource here is Openbook.fyi. If you’re not familiar, it allows you to search organizations and wealthy donors within the TESCREAL ecosystem. Very useful, and good to be aware of!)

Part III, The Outcome: This is my opportunity to distill all the criticisms I’ve made over the past 5 years targeting the TESCREAL worldview. I’ll argue that the ongoing attempt to create utopia is extremely dangerous. The reason is that “utopia” can justify virtually anything: if the stakes are sufficiently large — literally trillions of “digital people” in a sprawling multi-galactic civilization billions of years from now — then there’s no line that can’t be crossed; there’s no moral norm that can’t be violated.

Indeed, it’s this utopian promise that enables the evil AI companies to “justify” — if only in their own minds — the fact that they’re destroying the educational system, the rule of law, and the free press. Sure, things might get worse for most people in the near term, but that’s a cost worth suffering given the utopian “benefits” that await once AGI arrives.

A great video on how generative AI is destroying the educational system. Highly recommended:

I’ll also argue that if TESCREALists were to successfully create utopia, this itself would be utterly catastrophic for humanity. The argument here is that “utopia” is an inherently exclusionary concept. Someone is always left out of utopia — otherwise it wouldn’t be utopia.

So, the question one must always ask when confronted with utopian proclamations is: who is excluded from enjoying the paradisiacal delights being promised? With respect to the utopia of TESCREALism, humanity itself would be excluded. There is no place for our species in a world run and ruled by digital posthumans. This is precisely why I’ve been screaming about Silicon Valley pro-extinctionism for so long — and why I’m thrilled that folks like Taylor Lorenz are starting to talk about it. Many of these people really do want to replace humanity with AGI or “superintelligence,” not centuries from now but within our own lifetimes.

Part IV, The Revolt: Finally, this will outline what we can do to fight back against the tech oligarchs and the TESCREAL fantasies that motivate them. Surveys consistently show that the public is fed up with AI. In a sense, my anti-generative-AI side has already won the public debate. What’s missing is an organized movement that can direct all this anger and frustration toward reeling in and regulating the AI companies. There are already some movements working on achieving this aim, such as Stop AI. But, hopefully, my book will inspire more people to take a stand against the reckless race to build a Digital God.

Part IV will also explore alternative visions of the future. My friend Monika Bielskyte likes to say, “You cannot design for, you must design with.” I completely agree, and hence will argue that we need to ensure that representatives from every nation, culture, tradition, religion, demographic, and group has a seat at the decision-making table. (Here I’m reminded of Altman once saying: “Because if I weren’t in on this I’d be, like, Why do these fuckers get to decide what happens to me?” Exactly, dude.) I do not know what the future should look like, but I do know how to find out: by ensuring that everyone has a say through an inclusive and democratic process.

What do you think? This is basically a very short summary of Chapter 1, which I wrote last Wednesday in a burst of energy. I’m really excited about this project — although I’m also so freaking tired of thinking about the topic (!!). But it needs to be written. At this rate, I could probably have the book completed in two months. That’s not entirely implausible, as I wrote my 2017 book Morality, Foresight, and Human Flourishing in something like three weeks! There aren’t many things in life that I’m good at, but I am fairly good at becoming hyper-focused on, and monomaniacal about, writing projects. :-)

If you get a chance to read my Oxford Research Encyclopedia article, I’d love to know your thoughts on that, too. As always:

Thanks for reading and I’ll see you on the other side!