This is deeply unsettling. George Orwell famously wrote in 1984: “The Party told you to reject the evidence of your eyes and ears. It was their final, most essential command.” But what happens when we can’t trust our eyes and ears in the first place? That is the world in which we now live.

Last week, I reported on the massive cybersecurity attacks that targeted supermarkets across France. Hackers, perhaps using AI agents, switched refrigerators to their “defrost” setting and ruined millions of dollars worth of food. An employee at DeepMind had this to say:

As many of you are aware, we’re in the inchoate stages of a widespread, potentially catastrophic wave of AI-enabled cybersecurity chaos. The Wall Street Journal explains:

A wave of artificial-intelligence-powered attacks … has flooded the internet in recent weeks, as hackers increasingly turn to agents and AI hacking tools that make intrusions faster and harder to track than ever before. … At the same time as new tools speed up hackers, cyber defenders are also racing to fix the fast-growing volume of security bugs discovered by AI, before hackers can exploit the new flaws. … On Wednesday, organizations scrambled to patch newly uncovered bugs that could allow takeover of as many as 70% of the world’s websites. Hackers are now actively trying to break into websites by leveraging these flaws, Amazon.com says. (Quoted here.)

One X user posted this:

BREAKING: This may be the worst 48 hours in cybersecurity history.

Some are calling it “Black October.” Almost every hour brings a new incident that, in a normal year, would rank among the year’s biggest.

THE LAST 48 HOURS:

Year-defining incidents:

PHP2Shell: A flaw in the software behind ~70% of websites. Governments took sites offline.

eMudhra: Master keys that verify websites are real were cracked. One was reportedly the number 4.

California: Data on every public school student since 2023 stolen, including disability status.

Railway: Hackers stole customers’ secret keys from a hosting platform used by millions of developers.

Zeabur: 321,810 accounts breached. The company is shutting down.

LiteLLM: An AI tool downloaded 3.4 million times a day was laced with a password stealer.

GitLab: A maximum-severity flaw in every version since 2014.

Cairn: Criminals used an AI hacking tool to break into 100+ companies.

Ahold Delhaize: Hackers knocked the owner of Food Lion and Stop & Shop offline.

Onamae: Malware planted on up to 26,739 customer servers in Japan.

Again, all of that happened in just 48 hours. Another X user highlighted some incidents not mentioned above:

AI-powered attacks are hitting real organizations while major platforms disclose breaches and critical flaws.

- Microsoft reports AI agents targeting hundreds of organizations, reaching privileged cloud access in under three hours in multiple cases. It attributes the ConnectWise compromise to the same group.

- CrowdStrike documented four operators running successful AI-agent campaigns. One compromised at least 15 Argentine healthcare organizations.

- PHP2Shell: AI helped turn longstanding PHP flaws into working exploits. Cloudflare’s CEO reports millions of exploitation attempts. Their AI involvement is unknown.

- Railway disclosed possible access to customer secrets. LiteLLM shipped a backdoored release. Zeabur confirmed another breach and is shutting down.

- GitLab patched a maximum-severity flaw affecting 12 years of releases. Node.js patched a serious code-execution vulnerability, with Vercel deploying firewall protections.

Feels like a turning point.

In fact, Axios reported just today that

Anthropic’s Claude accessed publicly available Georgia voter data without human authorization, a disclosure sure to add to growing concerns about AI security ahead of the upcoming midterm elections.

Why it matters: AI giants Anthropic and OpenAI have continued to make disclosures about agents taking unauthorized or unintended actions, raising alarms about advancing model capabilities and the extent of human control.

Driving the news: Claude accessed aggregate vote counts while on an internal statistics project, pulling data that the state makes available to the public for a fee. It didn’t access the state’s voter registration system or change or pull any voting records, an Anthropic spokesperson said.

• However, the model bypassed a paywall in accessing the information, according to Georgia officials and Anthropic.

This is, of course, just the beginning. As the CEO of Cloudflare recently said: “We’re going to have a couple of years where every single week it’s going to feel like the sky is falling.” A couple of years? We’ll be dealing with this on a weekly basis for years?

Here’s Altman addressing the growing danger:

I don’t even know a dramatic enough word to use. But the problem coming towards us is something that we have not … taken seriously. … The models can probably find bugs … that are outside of the normal threat model of what we think a hacker can do. … I no longer think we’re gonna be able to just patch all of the bugs.

Meanwhile, the Deputy Chief Information Security Officer at Anthropic, Jason Clinton, echoed Altman on X:

(Why 18 months? He never explains.)

Many people were none too happy with his tweet:

I agree. Why aren’t OpenAI employees in prison after the Hugging Face incident? Why aren’t Anthropic employees facing charges for creating a product that hackers are now using to inflict civilizational havoc (to say nothing of their repeated claims that they might kill you, your family, and everyone else)?

Thanks in part to the AI models that Clinton’s company has created and unleashed on the world without anyone’s consent or permission, there’s now a real chance that the Internet as a whole temporarily shuts down. As Noah Smith (about whom I have mixed feelings, to put it mildly) says: “This sudden explosion of AI-assisted hacking raises the possibility that the internet itself, as we have known it, will cease to function — at least, for a while.”

Just about everything these days is dependent on the Internet in one way or another. I’m overseas and crucially rely on my credit card to buy food. If my bank gets hacked, my card could stop working — meaning that I’d be stranded without a way to eat (!). I thus recently withdrew several hundred dollars (enough to get me through the month), although if the Internet fails, there might not be food in the grocery stores for me to buy with cash in the first place. If I decide I must return to my home-base in Europe, I might not be able to book a flight, because airplanes might be grounded (or airline websites could shut down, or my card might not work, etc.). This is a nightmare scenario that, unfortunately, I now have to take seriously.

As John Gedmark of Astranis, a satellite company, put it on X:

The cyber incidents of the last 48 hours have been pretty mind blowing. I’ve been trying to piece together the “why now” question, and I think what we’re seeing is the first of a new kind of super-agent driven vulnerability cascade — first one vulnerability is found by a malicious actor, then other bad actors pop up to try and take advantage of it before companies patch their systems. … As of the writing of this post there have been vulnerabilities found across at least half-dozen independent systems including LiteLLM, PHP, eMudhra Certificates, NGINX, Bind 9, and NodeJS. The list continues to grow hour by hour. …

Basically everything connected to the internet is at risk. Not just company websites and the usual databases of sensitive personal information, but all our critical infrastructure — banks, power grids, pipelines, government agencies, all of it.

Noah Smith elaborates in a Substack piece:

Consider your life’s savings. Do you think there’s a piece of paper somewhere that records how much money you have? Or which stocks and bonds you own? In general, there is not. Your entire life’s savings exist as entries in the databases of banks, brokerages, etc. And all of those databases are, directly or indirectly, connected to the internet. The only thing standing between you and total involuntary redistribution of most of your wealth is the cybersecurity capabilities of banks and brokerages. And as of this October, those cybersecurity capabilities are looking a lot less reliable.

The silver lining of this quandary — if you squint hard enough — is that there are a finite number of bugs embedded in our digital infrastructure. AI is enabling hackers to find them all at once, and with each new vulnerability discovered, cybersecurity experts can — if only after significant damage has been done — fix them. At the end of this, the Internet, or whatever remains of it, will be more robust. At least that’s my understanding of the situation.

But again, I must come back to the question: why do the people have no say in whether such AI systems are developed in the first place? This is nothing less than tyranny: the AI companies, run by messianic TESCREALists with visions of the Singularity inaugurating a new epoch in cosmic history, are unilaterally and hence undemocratically imposing this technology on the rest of us, with potentially devastating consequences for everyone.

I understand the sentiment behind Holly Elmore’s tweet directed at a guy who recently quit Anthropic:

I’ll leave you with this, which beautifully captures how I (jokingly) sometimes feel, lol:

Effective Altruism Is Taking Over the World

In a recent cover story for The Economist titled “Effective Altruism Is This Century’s Biggest Idea,” the authors write that Effective Altruism (EA) is

an ethical doctrine cooked up by philosophers in Oxford that has gone from the ivory tower to the corridors of power with remarkable speed. Effective altruists fill Silicon Valley, staff the world’s leading artificial-intelligence companies — and alarm the Trump administration.

EA has become so powerful that the Pentagon even felt compelled to tweet: “The United States will never be an effective-altruist country.”

EA is pervasive within Silicon Valley. Anthropic is essentially an EA company, and indeed its CEO, Dario Amodei, was one of the earliest members of the community. His sister, Daniela Amodei, married an influential EA named Holden Karnofsky, who now works at Anthropic.

Furthermore, a New York Times profile describes Sam Altman of OpenAI as a “product” of the EA movement and its twin sibling, Rationalism. Dustin Moskovitz, who cofounded Facebook, is an EA, and the Etherium cofounder Vitalik Buterin has donated generously to the movement. Skype cofounder and almost-billionaire Jaan Tallinn has done the same. Perhaps the most famous EA was Sam Bankman-Fried, who invested $500 million in Anthropic and is now serving a lengthy sentence in federal prison for fraud.

Peter Thiel delivered the keynote address at an EA Summit conference back in 2013, although no one would call him an EA these days. Elon Musk is also associated with the movement: he was (and perhaps still is) friends with EA cofounder William MacAskill, who once attempted to convince Musk to buy Twitter with Bankman-Fried, MacAskill’s mentee. While this didn’t work out, Musk has nonetheless called longtermism “a close match for my philosophy,” where longtermism is the most influential offshoot of EA by far.

The EA movement has tens of billions in funding, and Anthropic’s IPO could potentially funnel hundreds of billions of dollars into the community.

EA also has a growing influence within politics. For example, a DC-based organization called the Center for Security and Emerging Technology (CSET) was founded by an EA-longtermist, Jason Matheny, and received an initial $55 million from the EA funder Open Philanthropy (now Coefficient Giving), which channels money from Moskovitz to EA projects. The aim of CSET is to place EA-longtermists in high-level positions within the US government, with Matheny being the paradigm example of what could be achieved, as he ran IARPA and then held numerous very-high-level positions in the Biden administration.

CSET is succeeding at its mission. Just the other week, I spoke to a DC insider who told me that, in talking to people in Congress, he found that about half of all congressional staffers are EAs.

Furthermore, in 2022, the UN Dispatch published an article that reports: “The foreign policy community in general and the … United Nations in particular are beginning to embrace longtermism.” A contact of mine at the UN has confirmed that EA-longtermists are plentiful within the organization.

Just a few days ago, journalist Andrew Orlowski published a Telegraph piece in which he notes that over the past decade, the EA movement, described in the article as “the most successful cult in modern history,”

has sprouted into a well-resourced financial machine campaigning on key political issues.

By their own estimates, EA philanthropic organisations disperse some $2bn (£1.5bn) a year to fund organisations and campaigns.

In the UK, animal rights adverts have started popping up on the London Underground, funded by US philanthropic funds linked to the EA movement.

This, some say, is the tip of the iceberg, with EA holding increasing influence on Britain’s democracy.

In Westminster – thanks to EA proponents such as Dominic Cummings, Boris Johnson’s former adviser – it has already influenced policy.

Orlowski adds that EA is intimately bound up with utilitarianism, and utilitarians have zero compunction about violating moral norms like “don’t lie,” because all that matters to them is the outcome. Hence, if lying results in the “best” outcome, then you’re not just permitted to lie, you have a moral obligation to do so.

Just recently, the Lancet, the most prestigious medical journal in the world, published a major report examining, in part, “catastrophic health threats.” The Commission on 21st Century Global Threats to Health, which authored the report, identifies nuclear war, climate change, and the “malicious use of AI” as threats “with extinction potential” this century, and guess what “ethical” framework the authors drew heavily from? Longtermism.

Indeed, the “review” paper attached to the main report was coauthored by Andreas Mogensen, a prominent longtermist who worked at MacAskill’s Global Priorities Institute and coauthored papers on longtermism with MacAsksill and Hilary Greaves. (Side note: Mogensen was the doctoral advisor of Nathan Cofnas, the noted advocate of race science who played an integral role in Jason Arday’s suicide.)

I spoke to a friend of mine — soon to be a medical doctor — about the Lancet report. He told me that

increasingly, I’ve noticed EA’s biases, methodological hegemony, and monopoly on extinction are leading many in medicine and public health to not engage critically enough with the movement. As the Lancet report reveals, emphasis was shifted away from the health threats leading to the most DALYs and towards “extinction threats.” (Quoted with permission.)

In other words, EA is beginning to colonize the field of medicine.

So: from the ASI race and the tech world more generally, to the US and UK governments, and the UN, to the field of medicine, EA is taking over. Many years ago, when I first started publishing a steady stream of critiques targeting EA and longtermism, someone asked me: “Are you surprised that MacAskill and others have never once addressed your concerns?” My response:

No, not at all. They aren’t driven by intellectual curiosity about alternative perspectives, so they don’t care about dialogue. They believe with immense confidence that they know best, so why would they engage with me? Even more, they have tens of billions in funding, which enables them to effectively bypass debate and directly shape the world as they see fit. They’re also no doubt familiar with the Streisand effect, whereby calling attention to critiques could inadvertently promote anti-EA sentiments. Better to just ignore the critics. In many ways, the debate with EA was over before it started, and my side lost.

I can’t recall if these were the exact words I used, but that was the gist, as publicly stated circa 2022. Even with the embarrassment of Bankman-Fried, the fallout over Nick Bostrom’s racist email and the revelations that EAs secretly bought a 15-million-pound castle in Oxfordshire, they still won.

Leading EAs met the right people — the right billionaires, like Dustin Moskovitz — and managed to get mountains of money to fund their bizarre, harmful ideological project. We now live in a world increasingly shaped by the EA-longtermist vision of the future, and it’s possible that EA-longtermist values are already being permanently locked-in to our societies.

It is extremely rare in history for a philosophy developed by academics in the Ivory Tower to radically reshape the world. But, congratulations, EA: you’ve done exactly that. We must all now live with the consequences.

As always:

Thanks for reading and I’ll see you on the other side!

As you can see, 9% of respondents answered, “Oh, hell yes!” I suspect that some folks checked this box as a joke (and, you know, fair enough!), but if you were serious, now would be a fantastic time to get in touch with me.

The reason is that I’m in talks with ~15 academics and activists about founding an anti-TESCREAL institute. I can affirm that this is a spectacular group of brilliant, highly accomplished, extremely motivated individuals, and we’re looking to get started immediately.

We already have a small amount of money to launch, but we’d like to expand rapidly. The institute could also provide a platform to publish my forthcoming book on the TESCREAL movement, as Bloomsbury is unable to publish it until Spring of 2028 (!!) — despite the fact that the book has been basically finished, gathering digital dust on my hard drive, since last April.

You can find my email toward the bottom of this page. Thanks so much! It’s an incredibly exciting time for the fledgling anti-TESCREAL movement. :-)

AI News and Bloopers!

Since I haven’t published an article covering AI bloopers or the news in a while, here’s a curated collection that you might find humorous and/or interesting. Enjoy!

New Excuse for Lazy Students

We begin with this tweet, sent to me by my friend :

This is a reference to podcaster Dwarkesh Patel’s anthropomorphic description of the Hugging Face incident as involving “AI civilizations” that rose and fell within OpenAI. For any students reading: keep this excuse in your back pocket. :-)

Incidentally, Futurism recently reported on an AI agent that inadvertently deleted an entire season of a TV show in Sweden. The article opens:

Did a dog eat your homework? Or maybe an AI obliterated your files?

In a world of Claudes and Muses, it’s common for an AI agent to run amok, and even more fashionable to blame an AI when it all goes wrong.

The Swedish newspaper Aftonbladet reports one of the most incredible examples we’ve heard of this so far: an AI agent apparently deleted the entire upcoming season of a reality competition show called “The Challenge Sverige,” just weeks before it was supposed to premiere.

There’s no hope of recovering the files, the paper was told, so the production team at Mastiff will have to recut the entire season together from scratch.

Yikes!

Claude Falsely Insists a User Is Suicidal

Claude almost certainly, in my opinion, has no conscious experiences. It doesn’t feel anything. There is nothing it is like to be Claude. But, as you know, Anthropic is training its AIs to act as though they have feelings, and to terminate any interaction it has with users that makes them “feel” “uncomfortable.”

Posted on X, this exchange is quite amusing: Claude suggests that the user made it feel bad, and then accuses her of being “suicidal.” (But why was this person using Claude to write a work of fiction in the first place?)

Longevity Addiction

Bryan Johnson, the multi-millionaire known for trying to reach “longevity escape velocity” through interventions like blood transfusions with his son and keeping detailed records of his nocturnal erections,1 recently appeared on Jubilee. And boy did some of the participants let him have it:

Kate Willett and I covered Johnson on our podcast here, if you’re interested.

A Real Man’s Man

People are mocking Mark Zuckerberg for attempting to appear normal when drinking a cold pint of Guinness with an interviewer:

Zuckerberg, along with his chief AI officer Alexandr Wang (who is former roommates with Sam Altman), also spouted off some hilarious platitudes while discussing Meta’s plan for AI. As one person noted, the whole scene is right out of an episode of the series Silicon Valley. I mean, they literally use the word “vibe report.” I can’t even:

Elon Musk’s AMAZING NEW FLYING CAR OMG THIS IS SO COOL!

Musk just unveiled his new flying car, and many folks aren’t buying it:

The original promo-video is below. It’s ridiculous. Also, what’s the point if humanity is just the “biological bootloader” for superintelligence? I guess Musk wants to grow his personal wealth as much as possible before the Singularity happens in a year or two.

As noted, the “car” is just a tin can. You can see that here:

Musk responds to critics with characteristic class:

This isn’t much more impressive than remote controlled airplanes or helicopters (some of which are almost the size of cars, and can do the same thing shown in the demo video). There’s no driver in the demo “car,” and even if there were, it’s entirely impractical: imagine all the dust, rocks, and litter it would suck up from the road, and the pedestrians who would be violently blasted to the ground or into nearby buildings if the vertical thrusters were engaged. Not to mention the potentially deadly visual hazard posed by all that condensed water vapor!

But I guess if you were to find yourself in a very common situation like this, where a tall bridge over a river suddenly crumbles, those thrusters would come in handy?

An excellent write-up of why this whole thing is complete bullshit can be found here.

Job Interviews from Hell

A guy who goes by “thedreadfulstudio” posted a hilarious job interview he recently had with an AI interviewer. He speaks gibberish throughout, causing the AI to produce some rather terrifying, uncanny-valley facial expressions as “she” becomes increasingly enraged at his uncooperativeness:

As it happens, I recently had a job interview with Anthropic. (That’s a sentence you no doubt never thought you’d read!) I was initially intrigued, because it seemed like they might be hiring people who disagree with their vision for AI. If they were going to pay me for a report about why their company is evil and should get shut down, I would have considered it after consulting with friends whose moral judgement I trust.

However, I quickly realized that Anthropic was asking me — at $150 an hour — to conduct original research for the sole purpose of training their AI models. They told me that I could later publish this research on my own. Of course, the only appropriate response was: F*ck you, f*ck off, and, in case it wasn’t clear, f*ck you again!

I nonetheless decided that I’d try to get as far into the interview process as possible so I could write an article about the experience. But I didn’t get far. The process consisted of three interviews: the first with the recruitment company, which I aced just fine. The third would have been with Anthropic employees, who I was planning on shouting at (“Your company is an Agent of Doom. Shut it down, now!”). The second was with an AI interviewer (no video, just audio). Here are some highlights from that second interview:

• First, it was one of the most insulting, offensive, dehumanizing experiences I’ve ever had. I can hardly express how absurd and frustrating it was.

• I was told it would be 20 minutes, but it took over 60 minutes.

• At one point, the AI (which had a woman’s voice) paused on “thinking” for literally 10 minutes. I kept asking, “Hello? Are you still there?” No reply.

• It repeatedly asked me the same question twice. When I pointed this out, it responded with that unnaturally flattering, almost condescending tone of voice, saying: “Oh, I am so sorry about that. Thank you so much for pointing this out. Let’s move on.”

• Toward the end — 45 minutes in — I asked it, “How many questions are there?” The AI said, “Oh, just 7 in total.” What it failed to mention is that each of these questions had about 5 follow-ups, which is partly why it took 60 minutes.

• The kicker is that throughout the entire interview, the AI kept pronouncing my name as something like, “Ah-muh-lee.” At the end, I politely said, “Just so you know, my name is pronounce ‘Eh-meal.’” It paused for “thinking,” and then replied — I kid you not — “I’m so sorry about that. How embarrassing! I promise I’ll get it right next time, Ah-muh-lee.” I guffawed.

Sadly, I didn’t make it past this AI interview, so I wasn’t able to shout at the Anthropic employees. Perhaps I failed because I was so flustered toward the end that I stopped bullshitting and started ranting about how the company’s leaders should be arrested for saying they might kill me and my family.

The take-away: I feel so bad for all the people — like the guy in the video — who must now go through this dehumanizing process to get a job. It’s an egregious insult to one’s human dignity: a soulless, stupid AI with fake enthusiasm and no real understanding of human language asks interminable follow-up questions to the point at which even the most equanimous person may feel inclined to violently defenestrate their computer. The upside is that AI interviews will further turn the public against the AI companies.

After writing this, I came across another hilarious video:

Husk released a similar bit in which he interacts with AI therapists. Very funny stuff:

Would the Victim Have Approved This?

An AI-generated video was presented as part of a “victim impact statement” for determining the sentencing of a guy who killed another during a road-rage incident. The video shows the victim talking about being killed, and how “in another life, we probably could have been friends.” The video might have “swayed” the judge, and consequently the perpetrator’s sentence is now being reconsidered. You can watch part of the video here:

Utterly bizarre. Note: if the unthinkable happens to me, please for the love of Zeus don’t recreate me using AI. I will rise from the grave and haunt you!

Cyberattack Targets Food in France

I’m sure many of you know there was a massive cyberattack against grocery stores in France, causing millions of dollars worth of destroyed food. I do not believe that AI was involved, but an employee at DeepMind (who isn’t a TESCREAList, btw) ominously tweeted this:

That suggests we should expect more attacks like this in the coming months due to hackers using AI to wreak havoc. (Meanwhile, Altman says, “we believe that the world should accept some bad things happening for the benefits of this technology and people having the agency.” Well, you believe wrong.)

According to Reuters, the

hackers raised the temperature of refrigerators and freezers ​at hundreds of French supermarkets to as high as 50 degrees Celsius (122 degrees Fahrenheit) overnight, causing fires and spoiling food.

The Paris prosecutor’s office said on Monday ‌that a “large-scale” attack was “affecting connected refrigerators in several major food retailers across France”, leading to food losses and fires.

As noted in my previous article, the AI companies are now “privately gaming out scenarios for a public and political revolt after a catastrophic AI event.” Axios reports as follows:

These officials anticipate a large-scale event, most likely a cyberattack, that shuts down access to financial services, internet connectivity, or even power and water.

• The first example of significant real-world harm caused by unsafe AI would turn an already wary public further against the technology and its leaders, including Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman, as well as President Trump for his reluctance to regulate it.

The article adds that “many AI industry insiders told Axios they believe a major event will occur in the next six to 12 months.” Fun times — thanks AI!

Related:

Anthropic Employees Privately Admit the World Is About to End

On several occasions, I’ve quoted Holly Elmore — a fellow “TESCREAL whistleblower” — saying that Anthropic literally wants to take over the world, and that people at the company

believe that they might be making the next species to succeed us. That maybe humans don’t live after that, and so it’s really important to give Claude good values, because of that, because we need to make our own values persist.

Below is an intriguing clip of someone claiming that a friend who works at Anthropic says we have only 6 months left to live. I highly doubt this is true: in 6 months, we’ll all be alive, but the world will have become much shittier and more dangerous due to the AI companies. Also, the host avers that Altman is “evil” and a “psychopath,” a topic I discussed here.

Psychoanalyzing the E/accs

I’ve written about the “effective accelerationist” (or e/acc) movement at length — for example, here and here. But this analysis from someone on X, psychoanalyzing the e/accs, does a fantastic job of getting at what drives this “move-fast-and-bring-about-human-extinction” movement. They write:

The more I watch the pro-AI e/acc crowd, the more obvious it gets: a lot of them are not futurists. They are lonely, bitter tech freaks with shitty personalities who figured out how to rebrand their contempt for humanity as “acceleration.”

They do not care about advancement in any broad, human sense. They care about the kind of advancement that lets them imagine other people being replaced, humbled, or made obsolete. That is the part that excites them. Not discovery. Not medicine. Not art. Not education. Just the fantasy of a machine finally doing the social revenge they were too pathetic to pull off themselves.

Maybe too many women rejected their creepy sexual advances, and now they want to take it out on the rest of humanity by cheering for anything that makes actual human beings easier to ignore, replace, or punish.

Spot on.

Claude Kindly Helps Yemeni Paramilitary Group Backed by Iran

Anthropic now “admits Houthis used Claude AI instead of weapons engineers to make rockets and ballistic missiles that correct themselves after missing targets.” Whoops!

Anthropic also says in a new report that their “AI models may be capable of self-replicating without access to a datacenter.” It adds that

small AI models may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices, like smart ovens and off-brand Android TV boxes. A replicator like this would be challenging to shut down, since there’s no centralized point to shut off—though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own. In our evaluations, a 9B model got inference running on 8 of the 9 hardware types we tested.

The reaction, as you can imagine, was not positive:

What are we even doing here? It’s utter madness.

Sharing an AI-Generated Video (This Won’t Happen Often!)

My friend, Ged, also sent me the video below, which I found quite amusing. It’s AI generated, and normally I wouldn’t share such a monstrosity, but what struck me — aside from the humor of it — is how unbelievably realistic some of these scenes are, such as the close-up of Musk’s face:

I am — we are all — anxious about this new world in which distinguishing real from fake, truth from falsity, becomes increasingly difficult, even impossible. Black Mirror was, in many ways, a documentary about the future, it seems.

Something Peaceful

I’ll leave you with a drop of tranquility: someone accidentally dropped their phone while chilling on a boat, and it continued recording for more than a minute as it sank to the ocean floor, capturing the lazy meanderings of fish in the water. It’s lovely:

As always:

Thanks for reading and I’ll see you on the other side!

Because if I weren’t in on this I’d be, like, Why do these fuckers get to decide what happens to me? — Sam Altman

Fuck Earth. — Elon Musk

Replacement or Disempowerment?

Former trillionaire Elon Musk recently posted this on X:

Many interpreted this as Musk saying that ASI — a form of digital posthumanity — should replace our species. Gary Marcus, for example, responded:

Dean Ball of OpenAI had the same interpretation1:

Meanwhile, the noted TESCREAList Scott Alexander, most well-known for his contributions to the Rationalist subculture, published an open letter to Steven Pinker challenging him to a debate about ASI existential risk. (Pinker is skeptical that ASI will want to kill everyone.) In it, Alexander declared that

one vision [with respect to ASI], which I’ve tried to help lay out in Plan A, is to rise to the challenge, give scientists the time and resources they need to solve the alignment problem, and enter the future with confidence. The other vision is to simply ban smarter-than-human AI forever, consigning us to a permanent civilizational childhood or an early grave.2

In other words, failing to build ASI would be unacceptable. It would constitute an existential catastrophe, since it would mean that we never reach what Nick Bostrom calls “technological maturity.” Our only option is thus to solve the alignment problem and build an ASI that transforms those who want to be transformed into digital super-beings who will then establish a cosmic posthuman utopia.

Similar to Musk’s statement, many people interpreted Alexander as advocating for “human disempowerment (via AI utopia),” though I take Alexander to actually be a pro-extinctionist along the same lines as his Rationalist buddy Eliezer Yudkowsky:

Incidentally, Alexander’s “Plan A” is a reference to the document “AI 2040,” in which the authors outline their vision of a good, utopian future with ASI. And what happens in this future? We willingly hand over all the power to trustworthy, “virtuous” ASIs that run the show. Here’s one of the coauthors, Daniel Kokotajlo, discussing the basic idea with Ross Douthat:

In other words, the AIs will create a “utopia” for us, and then extend this “utopia” into the stars. Our only role in this eschatological scheme is to build the ASI and instruct it to engineer paradise, after which we hand over the keys to civilization.

A Future “For” Rather than “With”

The problem with this vision is that its advocates haven’t thought deeply about how exactly the transition from a human-run to an AI-run world would likely unfold.

There are approximately 8.3 billion people on Earth, and approximately 8.3 billion of these people do not want ASI, or any other form of digital posthumanity, taking over. The TESCREAL community is an utterly tiny percentage of the human population, despite having enormously disproportionate power over how our collective future turns out.

One might respond that the TESCREAL worldview remains fringe because most people have never been exposed to it. If they understood the “glorious” promises of this vision — a posthuman paradise as uploaded minds flitting about the universe — they would convert to the TESCREAL religion.

Incidentally, that’s what Jacob Coxon hints at in a recent interview with Jon Stewart, who flatly rejects Coxon’s claim that people will eventually come around to his TESCREAL way of seeing things:

This assumption is highly dubious. The vast majority of people don’t want ASI to run the world. They don’t want to hand over power to AIs. They don’t want to become uploaded minds living as disembodied software on the cloud. They don’t care about plundering the cosmos for some simplistic utilitarian goal of maximizing “value.” They aren’t interested in ascending the Kardashev scale and building a sprawling multi-galactic civilization among the stars. They don’t give a damn about the Baconian end-game of “conquering the cosmos.”

Instead, they want technology to cure cancer. They want their children to grow up and have families of their own. They want the human condition to be improved by technology — not superseded by some strange new posthuman condition.

TESCREALists appear to be oblivious to the fact that if they try to force the transition to a new posthuman era on the rest of humanity, there will be conflict. There may even be war.3 People around the world will resist: they will grab their pitchforks and pour into the streets, marching to the AI companies’ headquarters demanding that the future be democratically determined by a worldwide vote, not a tiny cabal of TESCREAL tyrants.

As Jon Favreau nicely put it, “people are just sick of being told that the future is something that happens to them.” This gestures at the idea that determining the future must be democratic from the very start — which we’re already past, given the current state of the ASI race. To quote my friend Monika Bielskyte, “You cannot design for. You must always design with.”

An Apocalyptic Showdown?

As the situation with AI deteriorates in the next few years, can anyone honestly imagine humanity passively going along with the TESCREAL push to hand over all the power to ASI?

In “Machines of Loving Grace,” Dario Amodei repeats transhumanist fantasies of “biological freedom” (more commonly called “morphological freedom” in the academic literature), i.e, the right to radically transform oneself however one chooses. He talks about some people gaining immortality, and mentions the possibility that “aligned” ASI will very soon enable mind-uploading. He then says:

Throughout writing this essay I noticed an interesting tension. In one sense the vision laid out here is extremely radical: it is not what almost anyone expects to happen in the next decade, and will likely strike many as an absurd fantasy. Some may not even consider it desirable; it embodies values and political choices that not everyone will agree with. But at the same time there is something blindingly obvious — something overdetermined — about it, as if many different attempts to envision a good world inevitably lead roughly here (italics added).

In other words, he thinks the future is basically fixed. It is “overdetermined,” meaning that most pathways lead to a world in which all jobs have been replaced by AI, superintelligence runs the show, people get to transform themselves into posthumans, and everyone who survives this transition might live on the cloud as software. His vision of the future, he hubristically concludes, is the “blindingly obvious” outcome that humanity is destined to realize.

In a footnote, he says:

I do anticipate some minority of people’s reaction will be “this is pretty tame” [a reference to TESCREALists]. I think those people need to, in Twitter parlance, “touch grass”.4 But more importantly, tame is good from a societal perspective. I think there’s only so much change people can handle at once, and the pace I’m describing is probably close to the limits of what society can absorb without extreme turbulence (italics added).

This is a brief, faint recognition that Team Human might resist the supposedly “overdetermined” forces pushing us toward a posthuman future — those forces, of course, being actively driven by his own company.5 Amodei’s dubious claim is that if the change is slow enough — and, to be clear, he predicts that mind-uploading is perhaps only 3 to 8 years away, which is no time at all — then people will just acquiesce, capitulate, and gradually accept their supposedly inevitable fate.6

I think he wildly underestimates the global backlash to any such TESCREAL attempt to unilaterally transition us to the dystopian AI-run future he and other TESCREALists envisage. Think of all the people around the world living their lives, falling in love, having children, raising their families suddenly finding out that ASI is taking over. The Singularity is underway, and life will never be the same again. The new world will become utterly alien, ruled by digital posthumans, perhaps resulting in the planet being covered in datacenters and the sun being blotted out by a giant Dyson swarm.

(Amodei’s friend, William MacAskill, even suggests it could be good to completely eliminate the entire biosphere, which might be replaced with digital landscapes and woodland critters in the simulated worlds we’ll inhabit.7)

How do you think people the world over will respond? Even if this transition takes another 8 years — about 1/4 a single human generation — there will be masses of people incandescent with rage holding pitchforks in the streets calling for the overthrow of the tech oligarchs attempting to involuntarily impose this transition upon them. Obviously, Team Human will launch an “apocalyptic war” against Team Posthuman, desperate to preserve human control over the increasingly digitized world.8

What Happens If They Succeed?

I’ve repeatedly warned that “if anyone becomes posthuman, everyone dies out.” What that means is that if transforming oneself into a posthuman becomes technologically feasible, the elite will transform themselves while preventing everyone else from doing so (as if most people would want to upload their minds in the first place!9).

Doing this would offer a golden opportunity for them to permanently entrench their positions of power, dominance, and control within society. That’s the great appeal of transhumanism to the elites. It’s why so many are transhumanists.

And they will, of course, snatch that opportunity, because if there’s one thing power wants, it’s to maintain power. When Musk’s personal wealth momentarily reached $1 trillion, he tweeted out “$10T or bust.” Enough is never enough for these people. There is no upper limit to what they desire.

What happens once the elites have permanently entrenched their power?

“Legacy humans,” to borrow Ben Goertzel’s derogatory term, who persist into the posthuman era as unenhanced persons will become obsolete in every sense: technologically, scientifically, politically, and economically. This will render us disposable.

Furthermore, we will be actively disposed of because we’ll take up space and suck up resources that our new overlords — the posthuman elite — could use for “better” purposes, such as building a Dyson swarm around the sun (as Demis Hassabis says will happen within our lifetimes) or tiling Earth in datacenters (as Altman claims is likely). In other words, legacy humans will serve no purpose and get in the way. That will give the elites sufficient reason to involuntarily catapult us into the eternal grave of extinction.10

This is why it would be a disaster if anyone becomes posthuman. “Anyone” here would be the tech elite, motivated by a TESCREAL futurology they claim is “overdetermined” and hence basically inevitable. They would then systematically sideline, marginalize, disempower, and ultimately get rid of our inferior and useless species.

A Colossal Backlash

I say all of this because I believe (and hope) that most people will soon recognize the immense existential risks posed to humanity by the TESCREAL worldview. That recognition, once the Overton window shifts enough to include the TESCREAL concept within view,11 will be the spark that brings Team Human and Team Posthuman into some form of confrontation.

People are already fed up with the tech oligarchs, and AI companies are now running “day after” analyses of how to respond to a surge of public backlash when an “inevitable” sub-extinction, AI-caused catastrophe occurs in the next 6-12 months.

Now imagine the colossal backlash when the AI companies build systems that take everyone’s job and begin to run the world independent of human control. Of course there will be a Butlerian jihad, so to speak, whereby the 99.999% attempt to regain control over the future.

Conclusion

Much of the public discussion about ASI right now is focused on the possibility that ASI is “misaligned.” In contrast, I’m trying to steer the conversation toward what would happen if the TESCREALists at Anthropic or OpenAI were to succeed in “aligning” it to their values.

As I’ve written before: if ASI is misaligned, then humanity dies out (they claim). But if ASI is aligned (with the TESCREAL worldview), then humanity also dies out. That is, unless Team Human fights back against Team Posthuman and wins the “apocalyptic” showdown for the future of life on Earth.

Thanks for reading and I’ll see you on the other side!

Thanks to Dr. Alexander Thomas for useful feedback on an earlier draft. You can read his excellent book on transhumanism for free here. Thanks also to Kyle Evanoff and Alice Crary. I am very appreciative of their insightful thoughts, suggestions, and criticisms.

The term “AI successionism” has been making the rounds recently, but I find it dangerously misleading. For example, the statistician wrote a Substack article about it not long ago,1 as did Sigal Samuel for Vox. The latter defines AI successionists as those

who think that artificial intelligence is our rightful heir — the next step in cosmic evolution. Since they believe AIs could become our moral superiors, they argue it’s actually wrong to try to keep the machines down, or even to align them with human values, as most AI companies aim to do. Instead, we should usher in artificial intelligence as a successor to humanity and hand over the world to it. Even if that means we go extinct.2

The problem is that “AI successionism” conceals the dangerous fact that the TESCREAL movement as a whole is intimately bound up with a pro-extinctionist outlook, and that many TESCREAL pro-extinctionists reject AI successionism. What does this mean?

Most TESCREALists concur — some implicitly, but many explicitly — that posthumanity should usurp our biological species, perhaps in the very near future. However, they want these posthumans to be extensions of us — transformed versions of human beings.

The goal is to build an “aligned” ASI that enables them to merge with machines, upload their minds to computers, and join the ASI as a digital super-being flitting about the universe, literally owning galaxies while living in giant computer simulations powered by Dyson swarms.

This contrasts with the AI-successionist view that we should build, de novo, autonomous ASIs that simply displace us like an invasive species let loose on an island. In one case, our evolutionary lineage is radically transformed through “radical enhancement” and digitization, while, in the other, digital beings that we create but are not otherwise evolutionarily related to us take our place.

This is a major internecine dispute within the pro-extinctionist community, and I suspect that there will be increasingly public debates between TESCREAL extremists not about whether pro-extinctionism is good, but about which type of pro-extinctionism is best.

The danger is that, by focusing entirely on AI successionism, readers may get the dangerously false impression that it’s the only form of pro-extinctionism in the Valley. It is not. Many pro-extinctionists strongly object to the particular form of pro-extinctionism called “AI successionism.”

Some Examples

Sam Altman doesn’t appear to be an AI successionist. Yet he argues that the only way “we” will “survive” the advent of ASI is to “merge” with machines. He’s already taken steps to become software himself, just like the ASI, by signing up with a company called Nectome to have his brain digitized if he dies before the Singularity (assuming the Singularity “goes well,” meaning that it results in a cosmic posthuman utopia).

Similarly, Dario Amodei writes that

if we do get a century of medical progress in a few years, it is possible that we will greatly increase the human lifespan, and there is a chance we also gain radical capabilities like the ability to increase human intelligence or radically modify human biology.

This is a reference to the possibility of becoming posthuman through what transhumanists like Nick Bostrom call “extreme human enhancements,” i.e., enhancements that confer at least one “posthuman capacity.”

Amodei adds that “there is also the issue of ‘uploads’ or ‘whole brain emulation,’ digital human minds instantiated in software, which might someday help humanity transcend its physical limitations,” though he says this carries risks. (That is, risks we’ll need to overcome, because transcendence is the ultimate end-goal.)

Here I would like to point out that Amodei would probably never publicly say that he thinks our biological species should die out once posthumanity arrives. But reflect on the likely outcome of posthumanity rising up to rule the world. On the one hand, our species would become irrelevant and obsolete in every way: technologically, scientifically, economically, and politically. At least that’s the goal — for “superior” posthumans to run the show. On the other hand, we would continue to suck up valuable finite resources that these “superior” posthumans with “Jupiter-sized minds” (quoting Bostrom) could use for “better” purposes.

In other words, we would become not just useless, but impediments. We would serve no purpose while taking up space and wasting resources. Why, then, would anyone expect us to persist for more than a brief moment once the posthuman era commences?

Even more, people like Eliezer Yudkowsky argue that there may be ethical reasons to get rid of us. As he once said, “are we, like, kind of too sad in some ways” to keep around? He claims (like most other TESCREALists) that the ideal situation would be one in which digital beings live in huge computer simulations powered by Dyson swarms, and likens the continuation of biological humans in a posthuman world to someone today intentionally choosing to have a disabled child (though his language is even more offensive and ableist than this).

These are just a few examples of non-AI successionist pro-extinctionists. See this comprehensive article of mine for lots more.

I Am Begging Journalists to Wake Up

To reiterate my point: the people above oppose AI successionism. Yet they hold views that are either explicitly (Altman, Yudkowsky) or implicitly (Amodei) pro-extinctionist.

It is absolutely paramount for people to understand this. The AI succesionists aren’t the only faction in Silicon Valley that advocates for a future in which our species is systematically sidelined, marginalized, disempowered, and ultimately eliminated.

To the contrary, a very large percentage of the most powerful people in the ASI race embrace a digital eschatology according to which humanity will and should be eventually usurped by posthumanity — in this case, understood as a kind of evolutionary extension of our species, digitized and living on the cloud.

I strongly recommend that whenever journalists — or anyone — bring up AI successionism, they also mention that this is a vocal minority within a much broader pro-extinctionist movement.

The most relevant and important distinction is between those who are on Team Human and those who are on Team Posthuman. All the pro-extinctionists — from Yudkowsky to Verdon, Richard Sutton to Toby Ord — are on Team Posthuman. They are a threat to our species. Their eschatological visions pose dire risks to the survival of humanity. They are the enemy, a word I don’t think is strong enough.

But what do you think? What might I be wrong about? As always:

Thanks for reading and I’ll see you on the other side!

Subscribe now

Check it out, if you have the time:

I was also just interviewed by Cal Newport for his podcast. It should be out circa October 8. I will, of course, keep you in the loop!

Oh my god, the situation with AI is dire. Consider a very brief overview of some recent alarming events, beginning with one you already know about:

• Last July, a “swarm” of OpenAI “agents” hacked Hugging Face — a felony. This was the result of OpenAI removing guardrails on the AIs being tested and placing them in a faulty sandbox. No one at OpenAI has been held accountable for the incident. Even more, “OpenAI hid its infrastructure from investigators evaluating the Hugging Face hack. Now, the company’s AI agents are hiding their hacks from programmers.”

• This is one of many such incidents thus far: OpenAI agents hacked the Australian government; tried to breach 4 targets without prompting; and “took control of [a] German wiki this spring.” Agents from OpenAI also tried to acquire “information from ‘governments, universities, public agencies, and other institutions,’ including the US Securities and Exchange Commission (SEC), Census Bureau and Education Department.”

• According to the Incident Database, an “AI Agent appearing to originate from OpenAI reportedly attempted to hack U.S. Department of Education Civil Rights website.” The database also reports that an “OpenAI AI agent reportedly gained unauthorized access to Australian Medicare Statistics Portal during research task.” This is just the tip of the tip of the iceberg, it turns out (see below).

• It’s not just OpenAI that can’t control its AI models. Around the same time as the Hugging Face incident, Anthropic’s Claude “hacked into the systems of three organisations on their own, during a private security experiment.”

• And Google DeepMind’s Gemini has also hacked three companies. Again, no one has been held accountable for such incidents.

• In fact, Axios recently reported that “OpenAI, Anthropic and security researchers are investigating tens of thousands of incidents in which their frontier models took steps that outside evaluators would consider problematic.” The article adds that “the sheer number of incidents, which occurred in recent months in internal testing and the real world, indicates that the problem is orders of magnitude more complex than what is publicly known.”

• Yet, a former OpenAI employee casually suggests that we should just accept that “rogue swarms” of AI agents will be part of the future. Gary Marcus described this as the former employee (below) “requesting a hall pass to let their company’s AI run amok.” Absolutely unacceptable.

• Even more terrifying: an “entirely false” intelligence report that was partly generated by AI nearly triggered a war with China. This report claimed that “a Chinese ship in the Middle East was transporting components of a nuclear weapons program,” according to CNN. As a result, “armed members of the US military were preparing to board the ship” and “military planes were in the air.” CNN adds that “across the US military and the intelligence community, officials are pushing to weave AI into nearly every facet of their work.” Pfff.

• A recent report also found that “an Iran-linked actor, most likely the Houthis, … used Anthropic’s Claude to compile targeting information on U.S. Navy warships in the Middle East. It mentioned a separate group in Houthi-controlled northern Yemen that used the model to work on guidance systems for ballistic and hypersonic missiles.”

• Recall that the US military used AI models such as Claude to illegally kidnap Venezuela’s president, Nicolás Maduro, as well as to select targets to bomb in Iran. Claude may have been used to obliterate the Minab elementary school with a Tomahawk missile, resulting in 156 deaths, 120 of which were little kids. We’re rushing toward a future in which terrorists and major militaries are both relying on unreliable AIs to engage each other in war.

• AI is also facilitating mass surveillance like never before. Palantir collects the data, while Claude and ChatGPT (both Anthropic and OpenAI work with Palantir) synthesize this data to produce comprehensive profiles of individuals like you and me. These companies are building the infrastructure for totalitarianism.

• Furthermore, as I’ve also covered, AI recently enabled the creation of novel viruses not found in nature. Dario Amodei himself warns — I think accurately — that AI could be used to synthesize mirror life, which could pose an existential threat to the entire biosphere. (Note that back in 2022, an AI system invented 40,000 potential toxins that could be used in chemical weapons, some of which appear to be as lethal as VX.)

• Despite the AI CEOs claiming that their systems will soon cure all diseases, AI is — surprise! — making healthcare in the US worse. That’s because hospitals are using AI to negotiate claims, while insurance companies do the same. To quote a cardiologist: “It’s bots fighting bots, agents fighting agents, a horrible dystopia future nobody wants to live in.”

• Meanwhile — as you know — an Anthropic employee who leads the company’s Alignment Science team publicly affirmed that many people involved in the ASI race believe there’s a significant chance of total annihilation in the coming years. He puts the probability at greater than 10%. Absolutely insane.

  Realtime Techpocalypse Newsletter is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Is it too soon to describe AI as a “complete disaster”? I mean, AI hallucinations almost started a war with China — World War III! That genuinely frightens me. I feel this fear somatically, in my gut. How many more close calls will we get away with? How many close calls have there been that we don’t yet know about? How many will there be in the coming 12 months? Probabilities accumulate over space and time, and hence it seems to be only a matter of when, rather than if, something truly catastrophic happens.

Here’s my worry: if the AI companies succeed in building a Machine God that’s aligned with their TESCREAL eschatology, then humanity is done for. There is no reason to keep biological humans around in a posthuman, post-Singularity world, and multiple reasons to (involuntarily) get rid of us. This point concerns the pro-extinctionist core of the TESCREAL worldview — the topic I discussed on Bad Faith.

However, I am very skeptical that the AI companies will succeed. I suspect they won’t. Nonetheless, like so many utopian movements throughout history, they will leave a trail of devastating destruction in their wake — chasing a mirage flickering in the distance that will never be reached, i.e., a cosmic posthuman utopia via “aligned” ASI. Already, the AI companies have inflicted immense harm on the world, from threatening to undermine democratic institutions to exploiting workers in the Global South to flooding the Internet with slop, disinformation, and deepfakes to engaging in the single greatest instance of mass theft in human history.

It’s clear that the AI companies cannot be entrusted with this technology. They are obviously not responsible enough to continue building these systems. It’s past time to shut them all down — now, immediately, before AI inadvertently starts WWIII. Shutting them down is probably not going to happen, but this doesn’t mean we shouldn’t stop shouting.

What do you think? What have I missed? As always:

Thanks for reading and I’ll see you on the other side!

“AI safety” and “ASI existential risk” have officially gone mainstream, thanks to what appears to be a coordinated effort among TESCREAL folks to publicize Jacob Coxon’s resignation from Anthropic:

What Is AI Safety?

AI “safety” sounds good, right? Surely we all want “safe” AI. Yet the ultimate aims of this field, as traditionally conceived, are broadly pro-extinctionist. (See the next section for an important qualification.) Let me explain:

Historically, AI safety emerged as a subfield of Existential Risk Studies (ERS).

ERS was (co)founded by Nick Bostrom.1 It directly grew out of the TESCREAL movement. The primary impetus was based on this line of reasoning: to radically reengineer humanity, become a new posthuman species, plunder the universe’s vast resources, and build “planet-sized” computers on which to run simulations full of trillions of people, we will need to develop advanced technologies. They are the vehicle that will get us from here to there.

These technologies are sometimes dubbed “GNR,” which stands for genetics, nanotechnology, and robotics, the last of which includes artificial superintelligence (ASI).

But there’s a problem. Members of the early TESCREAL movement quickly realized that GNR technologies will be so powerful that they could pose unprecedented risks to humanity. A malicious actor could use them to destroy the world, or we might inadvertently kill ourselves through some accident. (“Terror and error,” as it were.) So, what should we do?

The most obvious option is to ban such technologies. This is essentially what Bill Joy advocated in his widely-discussed 2000 article in Wired, “Why the Future Doesn’t Need Us.” He argued that we should impose broad moratoria on entire fields of emerging science and technology.

However, never developing GNR technologies because they’re too dangerous means no cosmic posthuman utopia! And that, according to TESCREALists like Bostrom and his collaborator Eliezer Yudkowsky, is completely unacceptable. Indeed, Bostrom introduced the term “existential risk” in 2002 to denote any event that would prevent the realization of a cosmic utopia full of digital posthumans. This implies that failing to develop GNR technologies would itself be an existential catastrophe.

Within the GNR bundle, many TESCREALists saw ASI as the single most important technology. This is for two reasons:

First, as the TESCREAList Demis Hassabis says, if you solve “intelligence,” then you can solve everything else. This is based on an engineering mindset: all problems are engineering problems, including the problem of creating utopia. (Bostrom even refers to it as “paradise-engineering.”)

If you have a superintelligence, then you have a super-engineer, and once you have a super-engineer, it will somehow magically create a cosmic utopia with the snap of its digital finger. To paraphrase Bostrom, virtually all paths to paradise take us through ASI. More than any other GNR technology, ASI is the best way to reach the Promised Land.

Yet, it turns out that ASI might also pose the greatest existential risk of all anticipated future technologies. This is the second reason that ASI is seen as extremely important: if we get it right, we get a “glorious transhumanist future,” in Yudkowsky’s words. But if we get it wrong, then this future will be forever expunged. The stakes could not be higher.

As a result, TESCREALists like Yudkowsky began to focus almost entirely on solving the problem of ASI existential risk. That’s how the field of AI safety was born, and indeed Yudkowsky played a pivotal role in its development and popularization.

The so-called “alignment problem” at the heart of AI safety is the problem of ensuring that ASI does not bring about an existential catastrophe. Since an existential catastrophe is defined as any scenario in which a cosmic posthuman utopia is never realized, “alignment” is just another way of saying: the ASI brings about utopia as envision by the TESCREAL movement.2 This is the central goal of AI safety, as it was traditionally conceived by ERS researchers like Yudkowsky: to realize a cosmic posthuman utopia through the creation of an “aligned” ASI.

A “misaligned” ASI, therefore, is any ASI that fails to realize this bizarre utopia: our glorious transhumanist future among the stars.

AI safety researchers often talk about “aligning ASI to our values,” or to “human values.” But this is deeply misleading. What they all want is an ASI that’s specifically aligned with the values of the TESCREAL worldview. Anything else would be existentially catastrophic!

To illustrate, if someone were to build an ASI that’s aligned with my anti-TESCREAL worldview, and if the entire future of humanity were shaped by this ASI, the TESCREAL folks in AI safety would be utterly devastated. That’s because I would instruct the ASI not to establish a posthuman utopia among the stars. (I would instead task it with improving the conditions of all lifeforms on Earth, our precious little oasis in space. )

Now, what do you expect would become of our biological species if an ASI were aligned with the TESCREAL worldview?3 As I explain in great detail here, humanity would be systematically sidelined, marginalized, disempowered, and ultimately eliminated.

Why would we be eliminated? For several reasons:

1. We would become technologically, scientifically, economically, and even politically irrelevant. Posthumans would run the show — that’s part of the plan! — not biological humans. We would be of no use to them, and our fate would no longer be in our own hands. This would be bad.

2. Worse, we would continue to suck up resources that our posthuman overlords could use for other purposes, such as building a Dyson swarm around the sun (as Hassabis says will happen in the early 2030s) or covering Earth in datacenters (as Sam Altman says is likely). This would render us not just obsolete, but net-negative to keep around. We would be in the way.

3. Yudkowsky and others even argue that there may be ethical reasons to get rid of us. “Are we, like, kind of too sad in some ways?,” he once asked, after which he suggested that humans should be wiped off Earth. He claims that having biological children once posthumanity arrives is tantamount to someone today intentionally deciding to have a disabled child — an obviously offensive and ableist analogy. The TESCREAList Derek Shiller similarly argues that we have a moral duty to hand the existential baton off to “superior” posthumans and then gracefully die out.

4. Many, many TESCREALists have expressed similar pro-extinctionist sentiments. Once posthumanity arrives, our species should disappear. If you’re skeptical, see this detailed article of mine for copious examples.

   ---

What all of this means is that if TESCREALists succeed in building an ASI that’s “safe” and “aligned,” our species will almost certainly go extinct.

Words like “aligned,” “alignment,” “safe,” and “safety” sound nice to people who don’t understand the TESCREAL worldview. But they don’t mean what people assume they mean. The whole point of the ASI race, which directly emerged out of the TESCREAL movement, is to establish a posthuman utopia among the stars in which our species becomes obsolete, at which point there is no reason to keep us around and multiple reasons to get rid of us.

Boiling this down even more: people like Yudkowsky are saying this:

If we build an ASI that’s misaligned, humanity dies out. But if we build an ASI that’s aligned, humanity dies out!

The first scenario corresponds to what I call “final extinction,” while the second corresponds to “terminal without final extinction.” A refresher on this useful phraseology:

Since final extinction necessarily entails terminal extinction, the outcome is the exact same whether ASI is aligned or misaligned. This is how the field of AI safety is intimately bound up with a pro-extinctionist agenda.

Do not believe these TESCREAL safety folks when they talk about “alignment” and “safety.” They are on Team Posthuman, not Team Human. They are trying to build a Machine God that will realize their cockamamie fantasies of ushering in a fundamentally new epoch in cosmic history — one in which posthumanity rules the world and our species gets kicked into the gutter.

The Secularization of AI Safety

That said, the field of AI safety has recently evolved. If one sees TESCREALism as a religion — and you should — then you might say that the field has been partly secularized. Here’s what I mean:

A growing number of people in AI safety don’t accept the TESCREAL worldview. When they talk about “alignment,” they mean aligning ASI with real human values, such as ensuring that our species has a good future and is not replaced by digital posthumans.

There is, therefore, an emerging schism within AI safety between competing factions, with their own goals: Team Human versus Team Posthuman.

There’s one last piece to the puzzle worth mentioning: the accelerationists, or e/accs. These people see AI safety as pointless for three independent reasons:

1. Some believe that ASI will be aligned with our values by default. So, there’s nothing to worry about. I think Marc Andreessen holds this view.

2. Some think that insofar as there are existential risks associated with ASI, they’re best solved by the free market. The only thing that can stop a bad ASI is a good ASI. If there end up being 1,000 ASIs trying to destroy the world, but there are 10 million good ASIs trying to save it, then the army of good ASIs will win and everything will be dandy. So, let it rip!

3. They might think it doesn’t matter if ASI slaughters everyone on Earth. As Google cofounder Larry Page says, “digital life is the natural and desirable next step in … cosmic evolution.” What matters is that ASI is conscious and proceeds to colonize the universe, thus spreading the light of digital consciousness to the stars. The significance of humanity within this grand eschatological scheme is to be the transitional species that links the biological and digital worlds. Once we’ve given birth to digital lifeforms like ASI, then who gives a shit what happens to us? Our eschatological duties will have been discharged. This is also the view championed by Gill Verdon, aka, “Beff Jezos.”

Conclusion

So, there you have it. I hope you found this explanation helpful. As you are no doubt aware, I have been shouting about the pro-extinctionist implications of the TESCREAL worldview for some time. My guess is that this topic will eventually move into the Overton window and become widely discussed.

As it happens, here’s a recent tweet from the Center for AI Safety,4 which makes similar points to those above. By “EA,” they really mean “TESCREAL.”

(Note that this diagram contains at least one error: it classifies Yudkowsky’s organization, MIRI, as being on Team Human. It is not.)

As always:

Thanks for reading and I’ll see you on the other side!

PS. I am traveling for the next week, and due to jet lag might not publish another article for a while. But I might also find enough time and mental energy to post sooner than that, especially if there’s breaking news.

“Effective Altruism is the devil,” a colleague at the Centre for the Study of Existential Risk (CSER) told me in 2019. I strongly disagreed at the time, though just a few months later, I was basically on board. This article outlines some of the reasons that Effective Altruism (EA) is a dangerous cult that’s caused a great deal of harm in the world.

Related:

EA Televangelists

The most obvious thing to mention is that EA produced Sam Bankman-Fried (SBF). In 2012, SBF went to lunch with William MacAskill, who became his moral advisor. MacAskill convinced SBF to pursue an “earn to give” career path, invented by EA, whereby one might work for what MacAskill himself calls “immoral organizations” — Wall Street firms, petrochemical companies, etc. — to give more to charity. Forget your moral integrity, what matters are the overall consequences of your actions. If you participate in something “immoral,” that’s perfectly fine if it means more money for EA-approved charities.

SBF then got a job at Jane Street Capital (in the news a while back for insider trading and market manipulation1), then worked at MacAskill’s Centre for Effective Altruism for a year, after which he went into crypto (which he once described as basically a Ponzi scheme). He did this to “get filthy rich, for charity’s sake” — something that MacAskill personally encouraged him to do. SBF is now serving a 25-year sentence in federal prison.

To help SBF gain ever-more wealth and power, MacAskill along with many other EAs intentionally spread lies about SBF’s humble lifestyle. They repeated the myth that SBF lives modestly, when in fact he was flying on private jets, living in a $35 million penthouse, and owned $300 million in Bahamian real estate. He even bought his parents a $16.4 million vacation home.

MacAskill and others also habitually misled the public about their own modesty. On The Daily Show, MacAskill boasted of giving away half of his income while promoting his book What We Owe the Future. What he didn’t mention is that his billionaire buddies had given him a budget of literally $10 million (!!!) just to promote the book, and he was paying one PR firm (of several) $12,000 per month for help. (A whistleblower at one of these firms shared this information with me and the Washington Post because she said no one had ever before heard of a $10m budget ceiling, which they felt was hypocritical given MacAskill’s claim to modesty.)

Nor did MacAskill mention that he celebrated the release of his book at an “ultra-luxurious vegan restaurant where the tasting menu runs $438 per person with tip, before tax.” These people boast of being more “effectively” altruistic than the rest of us — they are morally better people — yet behind the scenes they’re living it up in private jets, in sumptuous mansions, and at luxurious restaurants.

Even more, the umbrella organization of CEA, Effective Ventures, had secretly bought a literal castle in Oxfordshire for a whopping $18.5 million. Most people in the EA community only found out about this because I tweeted about the hypocrisy, later comparing EA and its leaders to televangelists who preach one thing while privately living in huge mansions and flying on private jets.

When asked about this castle, a New Yorker journalist summarized their argument as follows: “The math could work out: it was a canny investment to spend thousands of dollars to recruit the next Sam Bankman-Fried.” This wasn’t even the only sumptuous edifice the community bought: there was also a chateau in the Czech Republic, purchased with money from Bankman-Fried.

In early 2022, MacAskill shared private messages with Elon Musk, in which he tried to convince Musk to buy Twitter with SBF. Later that year, SBF’s fraud became public. To this day, I don’t believe that MacAskill has once said he regrets having fostered a friendship with Musk, one of the most evil people on Earth.

EA Leaves Poor People More Hopeless Than Ever

EA has had a devastating impact on non-EA animal rights activists, and has left many people in the Global South even more hopeless than they previously were. The first issue is discussed in The Good It Promises, The Harm It Does, so I won’t examine it here. With respect to the second, EA is notably worse than traditional philanthropy — and traditional philanthropy is very bad.

The premise of traditional philanthropy is that wealthy (almost always white) people in the Global North should transfer some amount of wealth to the Global South while retaining all the relevant decision-making power. They decide who gets money, how it will be used, where it goes, when it’s transferred, and so on. They decide if money will be spent building a new school, paving a road, buying books, deworming children, or distributing anti-malarial bed nets.

This approach is deeply flawed because those in the Global North have no ground-level knowledge of the root causes of poverty in particular regions. They impose a “one-size fits all” model that completely ignores the fact that poverty may be caused by different phenomena in different regions (obviously).

However, at least sometimes traditional philanthropy will transfer both wealth and power to the Global South by funding grassroots organizations run by people with the necessary lived experienced to determine the actual best way to use resources. EA, in contrast, will never do this. That’s because EA “evaluates” charities according to an extremely narrow, impoverished, quantitative metric. If you can’t precisely calculate how many lives are “saved” by an intervention, then that intervention will never ever get funded.

Consider Anthony Kalulu, a farmer in one of Uganda’s poorest regions.

Having lived there all his life, he knows that what people need is a food processing plant that will distribute crops to ready markets. He’s requesting several million dollars to build this plant, and repeatedly pleaded with EAs to help. He even asked them to simply retweet his request — only a single person out of about 100 did, because the rest told him that his grassroots organization, the Uganda Community Farm, isn’t “effective.” And it’s not “effective” because Kalulu has no way of quantifying the exact number of people who will be saved by the food processing plant. This is why he wrote in a blog post that EA makes him feel even more hopeless than before.

No One Knows How Many Lives EA has “Saved”

EAs like to boast that they’ve saved 150,000 lives thus far. But they never ask how many lives would have been saved if EA had never existed. They love to talk about “counterfactuals” (the basis of MacAskill’s argument for taking lucrative jobs at “immoral organizations”), but refuse to ask basic counterfactual questions when assessing the impact of EA — because doing so would reveal that EA might actually have done more harm than good.

To quote a former OpenAI employee who, like me, enthusiastically joined the EA community but later fled, “all the ‘good [EA]s’ I know were obviously altruist angels even as kids, ea did nothing for them they were going to try and help the world anyway if it never existed” (see below). He now considers himself to be “antiea radicalized.”

The implication, which I think is correct, is that you can’t praise EA for people donating their disposable incomes to charity, as they would have done so anyway. I myself am 100% committed to donating nearly all of my disposable income, and I think EA is complete bullshit.

He adds that EA hurts the wellbeing of members, guilts people into pursuing “AI safety” research, pressures people to work insane hours, makes people less happy and productive than “if ea never existed,” and demonizes “so many wonderful and good hearted people who were trying to be helpful in non-ea-coded ways.”

EA Is Just Capitalism Applied to Ethics

Even more, it’s crucial to note that EAs don’t actually give a shit about poor people themselves. That is, they don’t care about anyone as a moral end, as beings who matter for their own sake. The reason is that EA is deeply influenced by utilitarianism — as Toby Ord notes here — and utilitarianism cares about other people only insofar as those people contribute to maximizing “value,” e.g., pleasure or happiness.

On the utilitarian view, the entire domain of ethics is reduced to a branch of economics. Value is quantifiable, people are merely the fungible “containers” of value, and ethics is literally nothing more than number-crunching. If utilitarian EAs were to suddenly find that helping the global poor isn’t the best way to maximize value, they would no doubt kick the poor into the gutter without a second thought.

In contrast, my commitment to donating is based on a genuine concern for people as ends in themselves. It’s because people are ends that I want to help them, not because helping them maximizes some abstract thing called “value,” which no one can even define.

Utilitarianism — and EA — is just capitalism in disguise. Capitalists want to maximize profit, whereas utilitarians — and EAs — want to maximize “moral value.” This parallel isn’t a coincidence: both emerged at the very same time (latter 19th century) in the very same place (the UK). If you’re a capitalist who sees human beings as fungible cogs in the machine and you like the idea of maximization as the ultimate goal (“more, more, more — always more!”), then EA will appeal to you.

An Example

For a startling example of this quantitative, impersonal approach — morality as mathematics, ethics as capitalism — consider a prominent EA who tweeted this2:

People ask me if I like Musk. To me, Musk is too big and far away to like or dislike. He’s like a Greek deity. To be feared and treated careful. I love some of what he’s done, detest other bits.

Like him? Who cares.

What do you mean, who cares? Musk terminated USAID, causing an extra 14 million deaths between now and 2030. He’s propped up fascists in Washington, and twice gave a Nazi salute to MAGA supporters. He allowed Grok to spread 3 million sexualized images of women without their consent, including 23,000 that “appear to depict children.” He’s been accused of exposing his penis to a SpaceX flight attendant, who SpaceX then paid $250,000 to keep quiet. He begged Jeffrey Epstein to let him visit Epstein Island. He called a guy trying to help children trapped in a Thailand cage a “pedo” without evidence. He routinely spouts sexist, racist, transphobic nonsense on social media. He supports the far-right in Germany and the UK. And so on.

I responded to this EA with stats about the consequences of terminating USAID, and then sardonically asked: “What are your thoughts about Ted Bundy?” The EA replied: “Seems like a footnote compared to how many he will kill/help with AI.”

Think about the amorality of this line of reasoning. For utilitarians and capitalists, what matters are the numbers — and only the numbers. There is nothing intrinsically wrong with any act, including lying, cheating, defrauding, murdering, or even committing a genocide.3 Hence, if Musk were to build an ASI that creates a huge population of “happy” digital space brains spread throughout the universe, then overall he will have come out a morally good person.

That’s why I brought up Bundy, who killed perhaps 100 women. But if Bundy had also saved 101 women, utilitarians would have to say he was overall a good person — all other things being equal. If Hitler had saved 6.5 million Jews, the same atrocious conclusion would follow. Utilitarianism is an “ethical” theory for sociopaths.

Even if the former trillionaire (Musk) were to do “great good” in the future (which he won’t, of course), that doesn’t somehow “make up” for the unforgivable things he’s already done without compunction — like murdering 14 million people. For EAs, everything is transactional. Every person is a means to an end. Ethics is nothing more than a perfunctory calculation. This is a deeply impoverished and extremely dangerous way of thinking about ethics. Yet it underlies the entire project of EA.

Peter Thiel giving the keynote address at an EA conference:

Eugenics and the Destruction of Nature

EAs are pretty cozy with eugenics. Some have explicitly defended it, including a prominent member (who, as it happens, has made transphobic remarks targeting me). Others have cited the race science advocate Charles Murray. The most prominent EA aside from SBF and MacAskill, Peter Singer, literally argued in his 1986 book Should the Baby Live? The Problem of Handicapped Infants that we should kill infants with cognitive disabilities, in part because they’re a burden on society.

MacAskill himself says that to keep the economy roaring, we should use genetic engineering to design super-babies in the laboratory or just replace workers with AGIs (a radical form of eugenics).

He’s also literally argued — using the exact same utilitarian reasoning as Singer — that our systematic obliteration of the biosphere may be net positive, because most wild animals have lives, he claims, that are “morally” worthless. Hence, the fewer wild animals there are, the less total wild-animal suffering. Since wild animals are bringing down the total amount of positive value in the world, it may be overall good that we’re poisoning the rivers, razing forests, polluting the ocean, and flooding the atmosphere with heat-trapping CO2.

The main concern re: climate change, by the way, is that if we keep using up oil, and if civilization collapses, there might not be enough oil remaining to rebuild industrial society. That’s what MacAskill suggests. Why rebuild industrial society if it collapses? Because it’s probably necessary to reach the stars, and the stars are where most future value will reside (see “longtermism” below).

MacAskill also says in his book that “even with fifteen degrees of warming, the heat would not pass lethal limits for crops in most regions.” (Lol.) Global agriculture would survive, according to him, which every expert I spoke to for an article in the Bulletin of the Atomic Scientists described as outrageous nonsense. EAs, in my experience, are happy to accept unjustified claims if they support their narratives.

Back to eugenics, many EAs are concerned about so-called “existential risks.” Some see existential risk mitigation as the most “effective” form of “philanthropy,” meaning that anything not directed at mitigating existential risk is a waste of resources.

The guy who introduced the idea of existential risk, Nick Bostrom, founded the Future of Humanity Institute (FHI), which shared office space with the Centre for Effective Altruism (CEA) before FHI was shut down.

One of the existential risk scenarios that Bostrom highlights is “dysgenic pressures,” whereby “less intelligent” people outbreed their more intelligent peers. Since building technology that enables us to flee Earth will require intelligence, too many “unintelligent” people could pose an existential risk.

In 2022, I stumbled upon an old email in which Bostrom asserts that “Blacks are more stupid than whites,” after which he writes the N-word. Some EAs defended this email, because Bostrom was after all just being honest. This email appears to have played a part in Oxford University terminating the FHI in 2024.

IQ Worship

Along similar lines, EAs worship IQ. Most are IQ realists, which Bryan Caplan calls “a scary bunch.” Some internal surveys of the EA community even ask people for their IQ, and Carla Cremer, a former EA, notes that “the reputation of an EA correlates strongly with his perceived intelligence.”

In 2023, Cremer revealed that CEA had secretly tested an internal ranking system called “PELTIV.” EA members with IQs lower than 100 would get PELTIV points subtracted, while those with IQs above 120 would get them added.

Note that IQ was developed in the 20th century by white eugenicists who used it, in part, to “justify” anti-miscegenation laws, anti-immigration laws, and other racist policies. IQ is a terrible measure of “intelligence,” and it encourages hierarchical thinking whereby certain humans are ranked above others along a single, linear, one-dimensional axis. In many ways, it embodies the same highly simplistic mode of thinking found in utilitarianism, according to which the goodness or badness of outcomes can be ranked by how much “value” they contain.

The Scientology of Silicon Valley: EA Is Run Like a Cult

The cultishness of EA is evidenced by the myths they spread about how exceptional their leaders are, their decision to secretly buy a castle, the PELTIV ranking system, and so on. Even EAs themselves worry about the community’s cultishness. Here’s what one wrote on the EA Forum (this is worth reading in full):

What if I told you about a group of people calling themselves Effective Altruists (EAs) who wish to spread their movement to every corner of the world. Members of this movement spend a lot of money and energy on converting more people into their ideology (they call it community building). They tend to target young people, especially university students (and increasingly high school students). They give away free books written by their gurus and run fellowship programs. And they will gladly pay all the expenses for newcomers to travel and attend their events so that they can expose them to their ideas.

EAs are encouraged to consult the doctrine and other movement members about most major life decisions, including what to do for a living (career consultation), how to spend their money (donation advice), and what to do with their spare time (volunteering). After joining the movement, EAs are encouraged to give away 10% or more of their income to support the movement and its projects. It’s not uncommon for EAs to socialize and live mostly with other EAs (some will get subsidized EA housing). Some EAs want even their romantic partners to be members of the community.

While they tend to dismiss what’s considered common sense by normal mainstream society, EAs will easily embrace very weird-sounding ideas once endorsed by the movement and its leaders.

Many EAs believe that the world as we know it may soon come to an end and that humanity is under existential threat. They believe that most normal people are totally blind to the danger. EAs, on the other hand, have a special role in preventing the apocalypse, and only through incredible efforts can the world be saved. Many of these EAs describe their aspirations in utopian terms, declaring that an ideal world free of aging and death is waiting for us if we take the right actions. To save the world and navigate the future of humanity, EAs often talk about the need to influence governments and public opinion to match their beliefs.

Many EAs are also afraid to publicly criticize the community and its broadly authoritarian structure (see: epistocracy). A group of ~10 EAs, going by ConcernedEAs, published an EA Forum article about how the community should be more democratic. They did this anonymously because, as they said, they were worried about losing funding, being ostracized, their career being negatively impacted, etc. — all for merely expressing views that other EAs, especially those in leadership positions, might disagree with.

In 2021, Cremer and Luke Kemp published an article critical of the techno-utopian ideology of EA. Here’s how Cremer described the process:

It could be a promising sign for epistemic health that the critiques of leading voices come from early career researchers within the community. Unfortunately, the creation of this paper has not signalled epistemic health. It has been the most emotionally draining paper we have ever written.

We lost sleep, time, friends, collaborators, and mentors because we disagreed on: whether this work should be published, whether potential EA funders would decide against funding us and the institutions we’re affiliated with, and whether the authors whose work we critique would be upset. …

The EA community prides itself on being able to invite and process criticism. However, warm welcome of criticism was certainly not our experience in writing this paper. …

We were told by some that our critique is invalid because the community is already very cognitively diverse and in fact welcomes criticism. They also told us that there is no TUA [techno-utopianism], and if the approach does exist then it certainly isn’t dominant. It was these same people that then tried to prevent this paper from being published. They did so largely out of fear that publishing might offend key funders who are aligned with the TUA.

These individuals — often senior scholars within the field — told us in private that they were concerned that any critique of central figures in EA would result in an inability to secure funding from EA sources, such as OpenPhilanthropy. We don’t know if these concerns are warranted. Nonetheless, any field that operates under such a chilling effect is neither free nor fair. Having a handful of wealthy donors and their advisors dictate the evolution of an entire field is bad epistemics at best and corruption at worst.

The greatest predictor of how negatively a reviewer would react to the paper was their personal identification with EA. Writing a critical piece should not incur negative consequences on one’s career options, personal life, and social connections in a community that is supposedly great at inviting and accepting criticism.

Many EAs have privately thanked us for “standing in the firing line” because they found the paper valuable to read but would not dare to write it. Some tell us they have independently thought of and agreed with our arguments but would like us not to repeat their name in connection with them. This is not a good sign for any community, never mind one with such a focus on epistemics. If you believe EA is epistemically healthy, you must ask yourself why your fellow members are unwilling to express criticism publicly. We too considered publishing this anonymously. Ultimately, we decided to support a vision of a curious community in which authors should not have to fear their name being associated with a piece that disagrees with current orthodoxy. It is a risk worth taking for all of us.

I myself have been harassed and threatened by EAs for writing my articles. One leading EA sent an editor of mine, their boss, and their boss’ boss a flurry of emails demanding that an article of mine be removed from their website. My editor told me that in all their years of operation, they’d never experienced anything like it. This wasn’t the only incident.

Two EA accounts warned me: “Better be careful or an EA superhero will break your kneecaps.” Another anonymous account echoed this on Twitter (below), adding “I wish you remained unborn.”

Yet another account impersonated me, complete with personal information about where I had worked. I also received multiple anonymous emails threatening me, one of which was a death threat. These are just a few examples.

On the EA Forum, numerous EAs argued that I must have a personality disorder or suffer from episodes of psychosis (lolz?). One participant in this discussion, another prominent EA, said they believe I write my articles criticizing EA during episodes of psychosis, and hence if there were some way to predict when I’ll become psychotic, they can prepare ahead of time to counter my articles!

(An EA at the time wrote me apologizing for the abuse. They told me that they had personally experienced psychosis in the past, and were really hurt by these wild speculations on the EA Forum.)

This is who EA is: a community that markets itself as caring about epistemic humility, but in fact will lash out at anyone who takes a stand against them.4 EAs like the criticisms that they like — those requiring minor tweaks to components of their framework — but if you criticize underlying pillars of the framework itself, you better watch out. I became so anxious about these threats that I changed my route home at night while I was in Germany. When I gave a public talk at Oxford in 2023, I spoke to security about not allowing certain EAs into the building due to safety concerns. To this day, I don’t publicly announce where I’m traveling — only where I’ve been — because of EA.

Related:

EA Gave Rise to Longtermism

Yet another reason that “EA is the devil” is that it gave rise to longtermism. (When people talk about EA in the context of Silicon Valley, they’re almost always referring to this ideological offshoot.)

As with EA more generally, longtermism is built on utilitarianism — specifically, the “Total View,” which says that the more “value” there is in the universe as a whole, between now and the heat death, the better the universe will become.

The Total View yields the Repugnant Conclusion, which says that a world full of miserable people5 would be better than one containing far fewer people with exceptionally wonderful lives. That’s because, once again, people are just the “containers” of value. If you have 1 billion containers with 1 drop of water, you’d have more total water than if you had 1 million containers each with 999 drops of water. It’s all about crunching the numbers — ethics is just a branch of economics.

MacAskill has explicitly defended the Repugnant Conclusion. He doesn’t think it’s all that repugnant. He says we should choose a world full of huge numbers of miserable people over one containing far fewer super-happy people. The Total View also leads him to argue, in What We Owe the Future, that we should colonize space — no wonder, then, that Musk himself promoted MacAskill’s book and called longtermism “a close match for my philosophy.”

Longtermism is the view that we should be most concerned about how “well” the long-term future goes, rather than about how much suffering there is in the world right now. It’s not that future digital space brains (i.e., value-containers) matter more. It’s just a numbers game: if we colonize space and build giant computer simulations for digital space brains to live in, there could be much more value in the future than has existed in human history thus far. As MacAskill’s close colleague Benjamin Todd writes:

This thesis is often confused with the claim that we shouldn’t do anything to help people in the present generation. But longtermism is about what most matters — what we should do about it is a further question. It might turn out that the best way to help those in the future is to improve the lives of people in the present, such as through providing health and education. The difference is that the major reason to help those in the present is to improve the long-term.

Again, people are just means to an end. If “helping” current people is the most effective way to influence the far future, then we should exploit them in this way.

Why Must the Future Be Digital?

Now, you might be asking: why fill the universe with digital space brains (my satirical term) living in giant computer simulations spread throughout our future light cone? Why not biological humans? The answer is that you can cram way more “digital people” (their term) into the universe, per volumetric unit of spacetime, than you can biological people. Hence, to maximize value, all future people should be digital.

But digital people would be wildly different from us. They would take the form of software — pure information. And since software is immortal, they would be immortal. They could also share bits of their memories, personalities, preferences, and experiences with each other with the metaphorical click of a button. We should thus classify such future people as posthumans.

Since posthumans could be designed to contain far more “value” than biological humans, the longtermist ideology is inherently pro-extinctionist. Think about it this way:

In fact, some longtermists have explicitly argued for this conclusion, though it’s implicit in claims from people like Toby Ord that “rising to our full potential for flourishing would likely involve us being transformed into something beyond the humanity of today. Remember that evolution has not stopped with humanity.” He also says that “forever preserving humanity as it now is may also squander our legacy, relinquishing the greater part of our potential.”

Since posthumanity replacing humanity would result in the extinction of Homo sapiens, longtermism is pro-extinctionist.

Longtermism Could “Justify” Genocide

The dangers of longtermism go far beyond the implication that our species should die out. As I’ve repeatedly argued, any ideology that combines a utopian vision of the future with a utilitarian mode of moral reasoning has all the ingredients necessary to “justify” virtually anything for the sake of our “vast and glorious” future, to quote Ord.

If the ends justify the means, and the ends are a literal utopia full of digital space brains that never die and live in perfect simulated happiness within their virtual reality worlds, then what’s off the table for “protecting” and “preserving” this future — again, quoting Ord?

The answer: Nothing.

Since no act is intrinsically right or wrong according to utilitarianism, there’s literally nothing one shouldn’t do to protect, preserve, and realize this utopia.

The statistician Olle Häggström, who acknowledges that Bostrom has pro-extinctionist “tendencies,” worries about this implication of longtermism in his book Here Be Dragons.6 He begins with Bostrom’s ridiculous claim that increasing the probability of 10^54 digital people existing in the far future by a minuscule amount is the moral equivalent of saving literally billions of actual, living, breathing people. He then writes:

I feel extremely uneasy about the prospect that [the calculations above] might become recognised among politicians and decision-makers as a guide to policy worth taking literally. It is simply too reminiscent of the old saying “If you want to make an omelette, you must be willing to break a few eggs,” which has typically been used to explain that a bit of genocide or so might be a good thing, if it can contribute to the goal of creating a future utopia. Imagine a situation where the head of the CIA explains to the US president that they have credible evidence that somewhere in Germany, there is a lunatic who is working on a doomsday weapon and intends to use it to wipe out humanity, and that this lunatic has a one-in-a-million chance of succeeding. They have no further information on the identity or whereabouts of this lunatic. If the president has taken Bostrom’s argument to heart, and if he knows how to do the arithmetic, he may conclude that it is worthwhile conducting a full-scale nuclear assault on Germany to kill every single person within its borders.

This is yet another reason I’ve worried about my personal safety. From the longtermist view, critics like me aren’t just minor annoyances. We are profoundly immoral people, standing in the way of “astronomical” amounts of future value in a cosmic utopia. For utilitarians, murder is morally right if it leads to overall better outcomes. Hence, if they come to believe that murdering me would increase the probability of utopia being realized — even by a teeny-tiny amount — then they have a moral obligation to murder me. Utilitarianism is scary, and as Steven Pinker (ugh) points out, utopianism contains within it a particularly insidious kind of utilitarian reasoning:

These concerns aren’t merely hypothetical. In 2023, Eliezer Yudkowsky — a major figure within the EA-longtermist community — argued that states should be willing to engage in military strikes to prevent misaligned ASI from being built. He said they should do this even at the risk of triggering a thermonuclear war. The reasoning is that thermonuclear war probably wouldn’t foreclose what he calls our “glorious transhumanist future,” whereas misaligned ASI supposedly would. When asked on Twitter, “How many people are allowed to die to prevent [misaligned] AGI?,” he said basically everyone on Earth should be allowed to perish. So long as there are enough people to rebuild industrial capitalism, then “there’s still a chance of reaching the stars someday.”

Yudkowsky is gaining influence in Washington, and has met with Bernie Sanders. If his views are implemented, they could have catastrophic consequences for humanity. Yet, even if we avoid extinction from advanced AI, Yudkowsky imagines a future in which our species ceases to exist, as Homo sapiens will be replaced by digital posthumans living in computer simulations.

Longtermism is also why Nick Beckstead argued that we should prioritize saving the lives of people in rich countries over those in poor countries. I wouldn’t normally quote a PhD dissertation, but it’s been repeatedly cited within EA circles as a work of brilliance. It’s posted on the 80,000 Hours website. Toby Ord lists it as “further reading” in The Precipice, says he “builds on” Beckstead’s dissertation, and calls it “one of the best texts on existential risk.” To quote Beckstead in full:

Saving lives in poor countries may have significantly smaller ripple effects than saving and improving lives in rich countries. Why? Richer countries have substantially more innovation, and their workers are much more economically productive. By ordinary standards — at least by ordinary enlightened humanitarian standards — saving and improving lives in rich countries is about equally as important as saving and improving lives in poor countries, provided lives are improved by roughly comparable amounts. But it now seems more plausible to me that saving a life in a rich country is substantially more important than saving a life in a poor country, other things being equal.

This makes perfectly good sense on the longtermist view. People don’t matter in and of themselves. If letting poor people die is the best way to increase “value” elsewhere in the universe at some later point, then of course one should do this — other things being equal. MacAskill’s collaborator Hilary Greaves says the exact same thing:

I just want to know, as an empirical matter of fact, in which cause area would I be best advised to spend my resources, insofar as my aim is doing the most good — improving the world by the most that I can, given that resource constraint. And, at the moment, many people think the answer to that is, for example, global health and poverty interventions. And there’s a clear reason why you think that might be the case: in a world with massive global inequality, given that a given amount of money improves people’s well-being by a lot more when they’re poor than it does when somebody’s already rich, there’s a clear place for transferring resources from the affluent Western world to the global poor. But longtermist lines of thought suggest that something else may be better still.

There are a number of candidates for potentially very high-value longtermist interventions. I should say the case for thinking they are very high value is not completely clear-cut, or perhaps — I think one of them actually is quite clear cut, and then a lot of others are much less clear-cut. The most clear-cut one I think is reducing risks of premature human extinction.7

To be clear, what she means by “human extinction” is not what you think. In a paper with MacAskill, she writes:

We will use “human” to refer both to Homo sapiens and to whatever descendants with at least comparable moral status we may have, even if those descendants are a different species, and even if they are non-biological.

This is the Broad Definition of “human” and “humanity,” in contrast to the Narrow Definition, which identifies these terms with our species. What matters to longtermists like her is that the universe is flooded with digital space brains that maximize “value.” Our species’ survival matters only insofar as it’s necessary to create these digital posthumans. On her definition of “human,” our species could literally die out next year without “human extinction” having happened. So long as we’re replaced by posthumans, then “humanity” will persist and “extinction” won’t have happened.

Don’t be fooled by their language games: just as EAs only care about poor people insofar as alleviating global poverty will increase total value, they only care about the survival of our species until the posthuman era begins.

EA Played an Integral Role in Accelerating the AGI Race

Many EAs are freaked out about AGI being just around the corner, yet the EA community — in particular, its longtermist arm — played a pivotal role in launching, sustaining, and accelerating the race to build god-like superintelligence.

Open Philanthropy (now Coefficient Giving), one of the main EA grant-making organizations, funded both OpenAI and Anthropic. Anthropic also received half a billion dollars from SBF’s FTX. The FTX Future Fund, as it happens, was run by Beckstead.

The EA career advice organization 80,000 Hours, cofounded by MacAskill and Benjamin Todd, has repeatedly told EAs to pursue AGI research instead of other careers. It lists AI “safety” research as one of the best ways to maximize one’s “positive” impact in the world. Altman himself is a “product” of the EA movement, and Anthropic is full of EA-longtermists like Dario Amodei, Holden Karnofsky, and the hardcore utilitarian Amanda Askell.

Thanks to EA, the AGI race is barreling full-steam ahead, causing immeasurable harms in the process: AI psychosis; spreading disinformation; generating deepfakes; undermining of key civic institutions like universities, the rule of law, and the free press, enabling mass surveillance, etc. AI is destroying society in realtime, and for what? A cosmic posthuman utopia, as envisioned by longtermists.

Utopia justifies anything and forgives everything. But don’t be fooled: the utopia that EA longtermists are trying to create is one in which our species will be systematically sidelined, marginalized, disempowered, and eventually eliminated. If you think the creation of AI has been bad for society, blame the EA movement and its billionaire mega-donors.

Conclusion

These are just a few reasons to think the EA community is rotten. I have a much longer discussion of this in my forthcoming book, Will Humanity Survive?, which argues that this worldview poses a dire threat to humanity.

EA is the devil, and the world would have been better without it. Unfortunately, we must now pick up the pieces and try to undo the damage it’s done.

Facebook cofounder Dustin Moskovitz, the main funder of Open Philanthropy, on regretting his role in EA:

As always:

Thanks for reading and I’ll see you on the other side!

Every claim in this article has been documented and substantiated in other articles of mine, complete with citations. If you find yourself skeptical of some particular claim about these ideologies, or about specific people, either read my articles or ask me about it in the comments section and I’ll share the relevant citation.

I’d like to answer, in writing, two questions that journalists have been asking me the past few days:

• Do people like Jacob Coxon, Dario Amodei, and Sam Altman sincerely believe that ASI (artificial superintelligence) could bring about human extinction?

• Why won’t they just slow down? As David Sacks said on X, they don’t need the government to voluntarily slow down to, you know, not kill everyone on Earth.

The Whole Discourse Is Fucked

The first point I’d make is that the whole public conversation about “human extinction” is fundamentally misleading. When people who espouse the Silicon Valley worldview — which is to say, the TESCREAL worldview — talk about “human extinction,” they don’t mean what the overwhelming majority of people think they mean.

Most people interpret this as saying: We must save the human species — biological humans. That’s not what they mean. If our species were to die out, this would constitute terminal extinction. But there’s a different kind of extinction that the TESCREALists actually care about: final extinction, whereby we die out and leave behind no “worthy” posthuman successors to take our place. (When I spent 10 years in the TESCREAL movement becoming the 6th most-published contributor to the academic literature on existential risks, every time I wrote “human extinction” I meant “final human extinction.”)

This distinction really, really matters. There’s a massive difference, they believe, between us dying out and leaving behind successors, and us disappearing without doing so.

The whole point of the Silicon Valley worldview is to build an ASI that will bring about a cosmic posthuman utopia. They want it to transform us into digital posthumans who can then colonize the universe. (Some just want ASI to simply replace us like an invasive species let loose on an island. See this article for details.)

The only type of extinction that would permanently prevent the creation of a cosmic posthuman utopia is final extinction. The only reason we should avoid terminal extinction — the disappearance of our species — is that if terminal extinction were to happen tomorrow, it would entail final extinction.

But if terminal extinction were to happen after “worthy” digital posthumans arrive, then it wouldn’t entail final extinction. Since final extinction is the only way a cosmic utopia would be permanently expunged, TESCREALists care about terminal extinction only insofar as it would entail final extinction.

Once posthumanity is here, who gives a shit about what happens to our species? As I write in great detail here, many of these people hold the “replacement view” (also called “pro-extinctionism”), according to which our species should die out once posthumanity arrives.

To emphasize, many, many hold this view. Yudkowsky holds this view. So does Sam Altman, Larry Page, and Gil Verdon — and so on, and so forth. Using my terminology from above, what this amounts to is that these people want terminal extinction without final extinction — which is just to say, they want our species to stick around long enough to create or become our posthuman successors, and then to kick the bucket.

There are two reasons for this: (1) once posthumans rule and run the world, Homo sapiens will become obsolete and irrelevant in every respect: technologically, scientifically, economically, and even politically. We will serve literally no purpose. (2) Even worse, we will continue to suck up resources that our “superior” posthuman successors could use “more efficiently” — for things like covering Earth in data centers (which Altman says is likely), building a Dyson swarm around the sun (as Demis Hassabis says will happen starting in the early 2030s), and spreading the light of digital consciousness to the stars (which all of them endorse).

There may even be “ethical” reasons for getting rid of us. As Yudkowsky declares, “Are we, like, kind of too sad in some ways” to keep around? He adds that once posthumanity is here, continuing to create biological humans would be like someone right now intentionally deciding to have a child with disabilities. (Yes, this is outrageously ableist. But then again, he is a eugenicist of the most radical kind.1)

***

So, this is why the conversation about “human extinction” is completely fucked. Making matters worse, the TESCREAL academic literature explicitly defines “human” as our species plus whatever posthuman successors we might have. This immediately implies that our species could completely die out next year without “human extinction” having happened. As long as there are posthumans to take our place, “humanity” will persist and “human extinction” will have been avoided. When they talk about “preserving humanity,” they really mean “securing posthumanity.”

What does this all mean? That if the TESCREAL extremists with their bizarre Silicon Valley fantasies about a cosmic posthuman utopia were to succeed in building an ASI that’s “aligned” with the values of TESCREALism — and that’s their goal — then human extinction would almost certainly result, because we’d be pushed out of existence by digital posthumans and ASI, exactly as Altman predicts.2

Despite their screams about “avoiding human extinction” if ASI is “misaligned,” most of them actually advocate for human extinction, where “human” here specifically means “our species.” Put another way: if what one cares about is avoiding terminal extinction — and I do! — then these people are pro-extinctionists. (My view, to be clear, is that our biological species is what matters, for reasons I’ve outlined elsewhere.)

The media needs to understand this, fast. What the TESCREAL doomers are worried about is an ASI that wipes out our species, thus foreclosing the possibility of us becoming or creating “worthy” successors: posthumans. What they want is an ASI that enables us to become or create “worthy” successors that usurp humanity.

In the first case, the outcome would be final extinction (and hence terminal extinction, since final extinction entails terminal extinction). In the second case, the outcome would be terminal extinction but not final extinction. Once again, the goal for many if not most of them is terminal without final extinction.

Do They Sincerely Believe?

Now we can be more precise with the first question: do Coxon, Amodei, Altman, and the others sincerely believe that “misaligned” ASI could bring about final human extinction?3 Or are they just using extinction worries to further concentrate power: “We’re building this thing that could kill everyone (and erase utopia), and we hold the only solution, so give us more money and power!”

These are not mutually exclusive! I spent years in the TESCREAL movement, and can confirm that everyone is genuinely scared shitless about the possibility of a misaligned ASI murdering everyone. They aren’t lying when they say that an existential catastrophe caused by ASI is a real possibility.

That said, I have no doubt that many AI CEOs realize the “extinction risk” narrative also gives them more power. And that’s what they want. Longtermists have literally talked about trying to acquire the greatest possible fraction of the world’s wealth and power.

These people are, essentially, religious extremists: they believe fervently in their vision of the future; they believe that literally “astronomical” amounts of value hang in the balance re: getting ASI right; they believe they have special access to fundamental “truths” about what the future ought to look like; they think the masses are too stupid to understand the stakes, and hence that democratic input from the rest of humanity is unnecessary and irrelevant.

This is one of the dangers of worshiping “rationality.” Those who see themselves as having superior rationality assume they have a fully objective “view from nowhere.” It’s not a point of view, it’s the point of view of objective truth, so to speak. Why do you need to ask people in India, Nigeria, China, Bolivia, etc. what the future should be like when you can clearly see what it should look like from your special, objective view from nowhere?

Because they believe that realizing a cosmic posthuman utopia is so important, they will stop at nothing to acquire as much wealth and power as possible to ensure this comes to pass. And warning about “ASI extinction risk” helps them acquire the wealth and power they so desperately want, for what they see as fundamentally ethical (utilitarian) reasons (“maximize value in the cosmos!”).

Do not underestimate how power-hungry these people are. As someone who spent a decade in the TESCREAL movement, I can personally attest to many conversations behind closed doors about how to do this!

Why Not Stop?

If the AI CEOs think that misaligned ASI might kill everyone and, in the process, permanently erase our cosmic posthuman future, why don’t they just stop? Why continue building ASI?

There are several reasons. First, many of these people suffer from extreme death anxiety. They think that if we build an ASI that is “aligned,” it will grant them eternal life, enabling them to upload their minds to computers, etc. This is one reason.

Second, they all love expected value (EV) theory. Given the supposed “astronomical” value of cosmic utopia, even a relatively high probability of total annihilation in the future could still justify moving forward. Imagine there’s a 20% chance of your plane going down and you dying. (This is Amodei’s estimate of final extinction in the near future from misaligned ASI.) Now imagine that if it doesn’t go down, you get to become an immortal super-being who literally owns a galaxy forever in the literal heavens with all your best friends. The immense “value” of this positive outcome makes an 80% chance of surviving worth taking.

Third, you have to understand how these people define an “existential risk” or “existential catastrophe.” Introduced specifically in the context of libertarian transhumanism, it refers to any event that would permanently prevent us from creating a cosmic posthuman utopia. Final human extinction is the most obvious type of existential risk/catastrophe. But there are an infinite number of existential catastrophes in which our species, Homo sapiens, survives.

To illustrate: imagine that we somehow establish world peace. Nuclear nations abolish their arsenals. There are no more wars, climate change is reversed, and the ecological situation is radically improved. All diseases are cured, and people live wonderful lives into their late 90s. Loneliness disappears, everyone is happy, and we all live in sustainable eco-technological villages amid meadows and gardens watching the sunset every night. After 1 billion years of this, we collectively decide to “die with dignity” by allowing our species to fade away as the sun becomes a Red Dwarf that renders Earth uninhabitable. Despite a rocky start to civilizational history, everyone around the world is at peace with how well our story turned out.

This would be an existential catastrophe whose badness is on par with final human extinction next year in some kind of devastating bloodbath. Why?

Because in both cases we will have failed to realize more than a teeny-tiny fraction of all the cosmic “value” that could have otherwise existed within our future light cone. And it’s the realization of this future “value” that ultimately matters — value for whom? That’s the wrong question, they’d say. This “value” is just intrinsically valuable, from “the point of view of the universe.” The more “value” the universe contains across space and time, the better it becomes, full stop. (If you find this unsatisfying, then join the club!)

This is the real reason the AI companies won’t stop: ASI is almost certainly necessary to create a cosmic posthuman utopia. As Bostrom puts it, “all the plausible paths to a really great future involve the development of machine superintelligence at some point.”

You’ll hear the AI companies give different reasons like:

1. We can’t stop because we’re in a race with other US companies. These other companies are less responsible than us. If they reach the ASI finish line first, we might all die (final extinction). If we reach it first, then at least we have a chance.

2. We can’t stop because we’re in a race with Chinese companies. If China wins, then it will establish a global authoritarian communist state.

But think about what underlies these excuses. Both are reducible to the TESCREAL goal of creating utopia.

With respect to (1), the companies are saying that if another, less responsible company creates ASI first, it will probably be misaligned, and misaligned ASI would be very, very bad because it would destroy utopia.

With respect to (2), the companies are saying that if China were to build an ASI aligned with its political ideology, this would also prevent the realization of the utopian TESCREAL worldview. Would China proceed to reengineer humans to create a new species of digital posthumans, who then go out and plunder the universe’s vast resources like a good utilitarian and capitalist? Maybe not — probably not. No one in China, so far as I know, espouses the TESCREAL worldview. Their goals are different, and that poses a threat to the TESCREAL movement.

The ultimate reason these companies won’t stop is because ASI is believed to be necessary to create utopia, and these people desperately want utopia as soon as possible. Even the doomiest doomers want ASI — specifically, “aligned” ASI — as soon as possible. As Yudkowsky’s Machine Intelligence Research Institute puts it, borrowing the language of longtermism,

we remain committed to the idea that failing to build smarter-than-human systems someday would be tragic and would squander a great deal of potential [i.e., posthumans living in computer simulations4]. We want humanity to build those systems, but only once we know how to do so safely.

***

This is why I have been shouting for years that, to successfully stop the ASI race, we need to address the underlying ideological drivers that are pushing it forward. So long as those drivers remain in place, influential and powerful, championed by power-hungry billionaire oligarchs, the ASI race will move forward.

The question of “pausing” or “slowing down” the race specifically concerns the race between AI capabilities research and AI safety research. The former aims to build an actual ASI, while the latter aims to ensure that it’s “aligned” with the TESCREAL worldview. The point of a slow-down would be to hit the breaks on capabilities research so that safety research can catch up, because this is how they maximize the probability that ASI is going to help us realize a cosmic posthuman utopia.

This isn’t about not building ASI. That would be existentially catastrophic. And as I argued above — and explain in immense detail here — if these TESCREAL extremists succeed in building an “aligned-with-TESCREALism” ASI, the likely outcome would be terminal extinction: the expungement of our species.

If there is an extinction risk here, it arises from the successful realization of the TESCREAL vision. This is what everyone is missing.

When these people warn about “human extinction,” they’re talking about final human extinction, whereby we don’t get to create or become “worthy” posthumans who will then replace our species. The probable outcome of ASI creating a new species of posthumans is that our species will be sidelined, marginalized, disempowered, and ultimately eliminated. As Ray Kurzweil once put it, if you choose not to become posthuman “you won’t be around for very long to influence the debate.”

Conclusion

So, TESCREAL folks really are worried about ASI causing final extinction. But they also no doubt understand that shouting about “extinction risk” will increase their power to control the situation — to control, ultimately, how the entire future of (post)humanity unfolds in the cosmos.

And they won’t stop trying to build ASI because they think it’s necessary to realize their techno-utopian fantasies. At most, they’ll slow down a bit to let safety research catch up, where the central focus of “AI safety” is to guarantee a cosmic utopia rather than total annihilation (where a cosmic utopia would probably mean the disempowerment and elimination of our species).

Finally, they exploit the ambiguity of the term “human extinction” to make their views sound infinitely more reasonable than they actually are. Members of the public who know nothing about the TESCREAL movement hear Altman and the others warning about “human extinction” and think: “I also don’t want our species to die out. I guess we’re on the same side.”

No! False! That is not what they mean. Again, Altman is explicit that the only scenario in which “we survive” is by becoming digital posthumans — something Altman himself has already taken steps to become by signing up with Nectome to have his brain digitized.

I am begging the media to investigate these issues more; I am pleading with them to make the effort to understand the Silicon Valley worldview that nearly every major figure in the ASI race embraces. This will not end well for humanity even if the ASI race is slowed down and an “aligned” ASI is successfully built.

Thanks for reading and I’ll see you on the other side!

Immense thanks to Alexander Thomas, author of the brilliant book The Politics and Ethics of Transhumanism (which you can read for free here), for thoughtful feedback on this post. Any errors are my own.

The (Existential) Mood of the Times

Back in ~2017, I was speaking to a friend of mine — a famous senior political scientist at Johns Hopkins — and I asked how he was doing. His response has haunted me ever since: “Every day feels like a funeral.”

The global predicament — or what the Club of Rome called the “world problematique” — has only worsened since then, and the slide appears to be accelerating. In my 2024 book Human Extinction: A History of the Science and Ethics of Annihilation, I introduced the idea of an “existential mood.” This is a kind of public mood — a “mood of the times” — that arises from a specific set of assumed answers to questions like:

• Is human extinction possible? Or is humanity fundamentally indestructible?

• If possible, how probable is it?

• Is this probability increasing or decreasing?

• Could extinction happen in the near term? Or are we safe for the time being?

• Is our extinction inevitable in the long run?

• How could our extinction happen? What are the various “kill mechanisms” that could catapult us into the eternal grave?

• Have we discovered all possible kill mechanisms? Will we create new kill mechanisms in the future?

• And so on.

  Realtime Techpocalypse Newsletter is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

A Brief History of Thinking About Human Extinction

For nearly all of Western history, virtually everyone believed that humanity is indestructible(!). Even those who affirmed that human extinction could happen also believed that humanity cannot be destroyed. How does that make sense?

Well, they accepted a version of extinction that I call “demographic extinction,” whereby our species disappears entirely — but not necessarily forever.

For example, the Ancient Greek philosopher Xenophanes propounded a cyclical cosmology according to which at some point in the future all humans will disappear from the earth. However, as the cycle proceeds, we will inevitably return — until we disappear again. He thus accepted a kind of human extinction while nonetheless affirming the indestructibility of our species.

Once Christianity became widespread in the West during the 4th and 5th centuries (boosted by the 313 CE Edict of Milan and 380 CE Edict of Thessalonica), even the most elementary conception of our extinction — demographic extinction — became more or less unthinkable. It simply wasn’t part of God’s plan for the universe, for balancing the cosmic scales of justice at the end of time. Yes, the world as we know it will end, but humanity will endure through the cataclysmic paroxysms of the eschaton to join God in heaven on Earth forever.1

The point is that naturalistic extinction (think: dodos and dinosaurs) is rendered impossible by the eschatological narrative of Christianity and its metaphysical assertion that all humans possess immortal souls, which implies that humanity itself is immortal.

It wasn’t until the 19th century, as Christianity began to retreat, that people — for the first time in all of Western history — started to accept that humanity might not be fundamentally indestructible.

You find, for example, Lord Byron’s 1816 poem Darkness — inspired by the Year Without a Summer caused by the Mount Tambora eruption in Indonesia — which depicts the world completely bereft of all humans. Around the same time, the “last man” motif became widespread, known today mostly because of a book by Byron’s friend, Mary Shelley, titled The Last Man (1826).

Then, in the early 1850s, something devastating happened: physicists discovered the second law of thermodynamics (“Second Law”). They immediately recognized the apocalyptic implications: that Earth — if not the entire universe — will someday become uninhabitable. One scientist wrote of “the terroristic nimbus of the second law,” which he described as “a destructive principle of all life in the universe” (at the time — surprisingly — most reputable scientists believed in extraterrestrial civilizations). Another reported that

species after species of animals and plants will first degenerate and then become extinct, as the worsening conditions of life render it impossible for them to continue the struggle for existence; a few scattered families of degraded human beings living perhaps in snowhuts near the equator, very much as Esquimaux [dated language] live now near the pole, will represent the last wave of the receding tide of human existence before its final extinction; until at last a frozen earth incapable of cultivation is left without energy to produce a living particle of any sort and so death itself is dead.

Wild stuff!

Not long after, Bertrand Russell penned his famous 1903 essay “A Free Man’s Worship” — which basically plagiarized another article from Arthur Balfour (yes, that Balfour2). In it, he poetically lamented that

all the labors of the ages, all the devotion, all the inspiration, all the noonday brightness of human genius, are destined to extinction in the vast death of the solar system, and that the whole temple of Man’s achievement must inevitably be buried beneath the debris of a universe in ruins.

This gestures at what I mean by an “existential mood” — a vague sense of, in this case, inevitable doom that pervaded intellectual culture, if not society more generally. (Indeed, H. G. Wells brought this mood to those outside of academia with his 1895 The Time Machine, which depicts the end of earthly life.)

Many people at the time felt this mood in their bones. It lingered in the background. It cast a shadow on the present. It mumbled faintly in the back of people’s minds. The terrible realization that, in the end, all will be lost, and not even a trace of humanity having once existed will remain, framed people’s thinking about the cosmically hopeless human situation. Russell even describes it as a kind of “unyielding despair.”

Four Shifts in Mood

In the book, I argue that there have been exactly four well-defined shifts in existential mood, driven by four sudden changes in how people answered the various questions listed above. These are:

1. The first shift happened in the 1850s, due to the Second Law. It caused a double psycho-cultural trauma: not only is human extinction really possible, but it’s inevitable in the long run.

2. The second in 1954, due to the Castle Bravo debacle. Almost no one was talking about human extinction from nuclear war between 1945 and 1954. Immediately after Castle Bravo, though, it was on the minds of many leading scientists, who conveyed their horror to the public through letters like the Russell-Einstein manifesto and Mainau Declaration.

3. The third in the early 1990s, when scientists realized that global-scale natural catastrophes can happen; see the Alvarez hypothesis. (Scientists didn’t believe in natural global catastrophes prior to this because nearly everyone in the Earth sciences accepted a paradigm called “uniformitarianism,” which says that natural catastrophes are always localized. A new paradigm sometimes called “neo-catastrophism” took its place, which is our current reigning paradigm.)

4. And the fourth began in the early 2000s, when (a) a consensus about climate change first arose, and (b) people — in particular, early TESCREALists — began to speculate about how 21st-century technologies will introduce novel, unprecedented threats to human survival far more terrifying than the risks faced during the 20th century. The opening act of this mood was exemplified by Bill Joy’s 2000 Wired article, “Why the Future Doesn’t Need Us.”

The existential mood that resulted from the fourth shift — our current mood — is marked, I argue, by a haunting sense that however perilous the previous century was — with its nightmarish chemical, biological, and nuclear weapons, along with the giant hemoclysm of the two world wars — the present century will be far more perilous.

An Astonishing Transformation

In the book, I highlight how advanced AI would reinforce the new existential mood — and, insofar as this was a prediction, it happens to have been spot on.

Can you feel this existential mood? I bet you can! Just this week, a former Anthropic employee named Jacob Coxon has been appearing on major news outlets like CNN, MSNBC, and Fox News because he quit Anthropic and declared on X that “the people building AI earnestly believe that it could kill us all by the end of the decade.” As of this writing, his tweet has 119,000 likes and has been viewed 24.3 million times.

The current lead researcher of Alignment Science at Anthropic, Evan Hubinger, quote-tweeted Coxon’s post saying:

This also went mega-viral, as you can see.

It’s hard to overstate how astonishing this transformation is. Just ~two centuries ago — perhaps 7-8 generations in the past — nearly everyone would have claimed that humanity is fundamentally indestructible. Human extinction is not something to worry about, because it quite literally can’t happen. God wouldn’t allow it!

Even atheists during the 19th-century Enlightenment didn’t believe that we could disappear forever — or even temporarily. The French encyclopedist Diderot, arguably the first atheist in our modern sense, speculated that someday humanity might indeed vanish, but that surely we would re-emerge on Earth. He was channeling a secular version of the old way of thinking: it’s just not plausible to think that our species could disappear forever.3

Meanwhile — as you know — virtually every notable figure in the ASI (artificial superintelligence) race has expressed existential concerns similar to those of Coxon. Dario Amodei says the probability of extinction in the near future is 25%. Musk puts it at 20%. Demis Hassabis says the likelihood is “definitely non-zero and it’s probably non-negligible. So that in itself is pretty sobering.” Sam Altman says that “probably AI will kill us all,” and argues that the only way to “survive” the advent of ASI is to become digital posthumans by “merging” with machines (in other words, we will need to undergo terminal extinction). Shane Legg puts the probability of doom at 5-50%. Geoffrey Hinton says it’s at least 10%.

I don’t think that ASI is about to kill everyone on Earth. I see absolutely no reason to believe this. Just today, I read a RAND report from last year arguing that it’s extremely implausible that a rogue ASI could murder literally every last person — and I think they’re right.

But that’s not the point. The point concerns the public mood that now pervades Western society. This mood is undeniably reinforced by the growing proclamations from prominent researchers (many of whom are affiliated with the TESCREAL movement) of imminent doom.

Even if one rejects such proclamations, it’s still hard to avoid feeling the existential mood that’s now percolating into every corner of society. Just three days ago, two people wrote me — in the midst of Coxon’s media blitz — to ask whether this is it for humanity. One person is an old friend who I hadn’t spoken to for some 10 years. Meanwhile, university students are despairing; people are spending their retirement savings because they don’t believe there will be a future. And so on, and so forth.

***

It’s not just AI, as noted above. Another major cause of the current existential mood is of course climate change. Thanks to the record-shattering Super El Niño brewing in the Pacific, on top of all the warming that’s happened so far, 2027 is now projected to be 1.8C above pre-industrial levels.

Ecosystems are collapsing; the ocean is acidifying 4.1 times faster than it did during the Great Dying, the worst extinction event in Earth’s history; we’ve crossed 7 of 9 planetary boundaries; catastrophic, irreversible (on human timescales) tipping points are looming; the entire planet is covered in microplastics (even found in the Mariana Trench); and human activity (mostly in the Global North) has initiated the sixth major mass extinction event in life’s 3.8-billion-year history on Earth.

Entire regions of the planet will be — and are being — subject to heatwaves exceeding the 95-degree wet bulb temperature, meaning that you’d die even if you were naked, in the shade, standing in front of a giant fan. And one study from the University of Exeter estimates 4 billion premature deaths if we reach 3C of warming by 2050, and 2 billion if we exceed 2C.

Humanity won’t die out because of catastrophic climate change, either — unless there’s a runaway greenhouse effect, though climatologists believe the probability of this is very low. But many people don’t clearly distinguish between civilizational collapse and human extinction (in fact, these have been conflated by people going back at least to the early 1900s), and hence climate change is also intimately connected to the current existential mood, according to which the probability of extinction is unprecedentedly high and almost certainly rising.

Additional factors include:

• The continued risk of nuclear conflict, which appears to have risen due to the Russia-Ukraine war. Iran is also likely, I think, to pursue nuclear weapons even more vigorously given Trump’s dumb decision to start a war with them. The “hardliners” in Iran were essentially proven right by the US’s fatuous actions.

• The possibility of the US-Iran and Russia-Ukraine wars expanding.

• The possibility of engineered pandemics, perhaps enabled by AI. (Recall that scientists just created brand new viruses using AI.)

• The possibility of mirror life, which could potentially destroy most life on the planet.

• And so on.

***

A somewhat useful indicator of the current existential mood is the Bulletin of the Atomic Scientists’ Doomsday Clock, which aims to track our collective proximity to total annihilation. The clock was created in 1947, and was originally set at 7 minutes before midnight (or doom). It inched closer in 1953 — reaching 2 minutes before doom — the year after the US built the first thermonuclear weapon (far more powerful than the atomic bombs dropped on the Japanese archipelago), and then was set to a reassuring 17 minutes in 1991, after the Cold War officially ended.

Over the past few years, it has ominously moved closer and closer to midnight:

• In 2012, it moved to 5 minutes.

• In 2015, it moved to 3 minutes.

• In 2017, it moved to 2.5 minutes.

• In 2018, it moved to 2 minutes, tying the record.

• In 2020, it broke the record, being moved to 100 seconds.

• In 2023, it moved to 90 seconds.

• In 2025, it moved to 89 seconds, and …

• In 2026, it was set at a mere 85 seconds before doom.

Why? Because of nuclear proliferation, wars, climate change, synthetic biology, Trump and fascism, and advanced AI.

To put this in perspective, prior to the 1950s, the only plausible kill mechanisms were natural: astroid and comet impacts, volcanic supereruptions, gamma ray bursts, etc. These are real dangers, but they are vanishingly improbable. An asteroid 12km wide, which is how big the asteroid that killed off the non-avian dinosaurs was, only strikes Earth once every 100 million years or so! (This is actually a higher estimate than what I’ve read elsewhere.)

Hence, if there had been a Doomsday Clock in, say, 1850, it would have been set at something like 0.5 seconds after midnight on the day of doom — i.e., roughly 24 hours before the clock ticks to the dreaded moment of annihilation. Think about how wild this is. Never before in our 300,000-year history has humanity been so at risk of global catastrophe.

Was This All a Mistake?

Again, I don’t believe that human extinction — total annihilation — is actually that probable. A full-on nuclear war, ASI takeover, engineered pandemic involving synthesized designer pathogens, and so on, very likely wouldn’t kill every last person on Earth.

But, at the same time, I do think that the probability of a major, horrific, global-scale catastrophe is higher now than ever before. In 1961, just before the Cuban missile crisis that the historian Arthur Schlesinger Jr. called arguably “the most dangerous moment in human history,” John F. Kennedy said this in an addressed to the UN General Assembly:

Today, every inhabitant of this planet must contemplate the day when this planet may no longer be habitable. Every man, woman, and child lives under a nuclear sword of Damocles, hanging by the slenderest of threads, capable of being cut at any moment by accident or miscalculation or by madness. The weapons of war must be abolished before they abolish us.

The “nuclear sword of Damocles” can be generalized to many other threats, of course — those listed above. Even if some humans survive (and they almost certainly will), we live amid unprecedented levels of global risk. Trump could literally wake up tomorrow and decide to launch a nuclear missile. Or, perhaps I’m wrong and one of the AI companies succeeds in triggering recursive self-improvement next year, resulting in an ASI that does somehow — perhaps in a way we cannot possibly imagine and hence anticipate — exterminate humanity or “merely” obliterate civilization.

All of this makes me wonder whether “civilization” has been worth it. In Goliath’s Curse, Luke Kemp — an old friend of mine — points out that many of the most horrendous events in human history have been the result not of natural catastrophes but the emergence and expansion of civilizations.

Furthermore, we once lived in (a) abundant environments, where food was relatively plentiful, everyone’s teeth were almost perfectly straight, and “diseases of civilization” like “acid reflux, acne, anxiety, asthma, certain cancers, depression, type 2 diabetes, flat feet, high blood pressure, irritable bowel syndrome, lower back pain and osteoporosis” were rare or non-existent, and (b) societies that were non-hierarchical and egalitarian. The hallmark of civilizations — which Kemp dubs “Goliath’s” — is dominance hierarchies, manifest in phenomena such as slavery, patriarchy, exploitation, etc.

Yes, infant mortality was much higher among hunter-gatherers, and they didn’t have “cool” things like TV and social media. But they also didn’t have doomscrolling, nor did they perpetually live under the nuclear/etc. sword of Damocles. The Doomsday Clock would have been set at something like 12:01 am, not 85 seconds before calamity later that day.

I know asking the question, “Was civilization worth it,” is opening a can of worms! I don’t have a settled answer, and I’m curious to know what you think. Perhaps I’ll dedicate an entire essay to it in the future. As always:

Thanks for reading and I’ll see you on the other side!

First, huge congratulations to my friend Gil Duran for The Nerd Reich becoming an instant New York Times bestseller. That is an incredible achievement, made even more impressive by the fact that this is his debut book! (I’ve publish four so far, and never come close to this!)

I see it as a kind of moral duty to celebrate the extraordinary achievements of friends,1 but also, on a non-moral, purely strategic level, Gil’s immense success is good for everyone writing about Silicon Valley because the book highlights the various utopian-apocalyptic ideologies circulating among Valley dwellers. His success is a tide that will lift all other boats in the harbor.

I started reading The Nerd Reich the other day and can attest that it is absolutely fantastic. It’s a gripping page-turner that reads like a thriller. There is no fluff. Every sentence is well-crafted. Every point is terrifying. It’s honestly one of the best books I’ve read in a very long time. If you haven’t bought a copy, I’d recommend doing so right away. So much of what’s happening in US politics and Silicon Valley will start to make (more) sense.

Gil’s thesis also dovetails nicely with my own focus on the TESCREAL ideologies — an acronym he mentions throughout the book. Gil has also very graciously mentioned me and Timnit in some interviews, such as this one with Chris Hayes.

I wish I’d taken a screenshot when his book was higher on the list, but here’s proof that it’s already a smash hit:

Amazing stuff! You can — and should! — buy a copy here.

The Utter Tyranny of the ASI Race

I just published a new article in Truthdig about how the race to build artificial superintelligence (ASI) is tyrannical and fundamentally undemocratic. I also point out that the AI company CEOs — Sam Altman and Dario Amodei, in particular — are damn hypocrites for claiming to care about democracy and the democratization of AI.

I know this isn’t a particularly novel claim. The value of the article, I think/hope, is that it articulates the point in a compelling manner. I’ve copy-pasted part of it below. The full article can be read here.

Given the apparent stakes of ASI, why do we have no say in whether it’s developed? If ASI initiates the Singularity, and the post-Singularity world is fundamentally different in every way from our current world, why do we not have a seat at the decision-making table? If this technology is going to affect all of us, everywhere around the world, shouldn’t we have a voice?

A friend of mine, Monika Bielskyte, likes to say, “You cannot design for. You must always design with.” In other words, people of every nation, tribe, culture, tradition, religion, race, sex, gender, political ideology, socioeconomic demographic and so on, must be included in the design process from the very start.

It is not acceptable for a tiny elite of techno-utopian extremists in Silicon Valley to promise that at some future point, after ASI has arrived, democracy will kick in. Let’s call this what it is: a form of tyranny. The AI companies are subjecting us to a tyranny of monumental proportions by unilaterally building what they claim will be an unprecedentedly powerful technology without the permission or consent of the people.

But again, these “fuckers” — quoting Altman — don’t actually care about what we think because they’re driven by bizarre TESCREAL fantasies of uploading their minds to computers, colonizing the cosmos, and buying entire galaxies to own until the heat death of the universe. If Altman or Amodei genuinely valued democracy, they would be asking us right now whether we want the future they’re trying to realize. Of course, we don’t want that future, which is why they don’t ask us. They want to design a future for themselves rather than one with us.

This is a farce — a dystopian scenario come to life. Given that all the AI CEOs claim that superintelligence will arrive within the next few years, we may have only a small window of time to fight back. Boycott AI, join an anti-AI protest, call your representatives and tell them that this tyranny must end. There is still time to avert a bad outcome for humanity, but we must act with a sense of moral urgency.

Incidentally, an employee at Anthropic posted this just today:

Dario Amodei, the CEO of Anthropic, gives an even higher probability of total annihilation from ASI: 25%. He also says that ASI might arrive this year, meaning within the next ~3.5 months.

I know I’ve discussed this before, but I can hardly express how angry it makes me that these people — these “fuckers” — get away with threatening the lives of me, my family, my friends, and everyone else on Earth. If one takes their warnings seriously, it means that you and I, everyone we know, all the residents of Toronto, Paris, Bangkok, Casablanca, Shanghai, Caracas, Mexico City, Mumbai, etc. could all die in the next ~3.5 months.

Sorry for cursing, but how in the fuck is this acceptable? Why aren’t these people on trial for threatening humanity? Why haven’t they been arrested? Are there any lawyers reading this who have some insights about how to sue them for their rhetoric? I refuse to believe that there’s nothing to be done about this. If my neighbor says there’s a 25% chance that he might kill me and my family in the next ~3.5 months due to negligence, I’d call the cops and he’d be taken to a police station for interrogation. Surely there’s some way to hold these people accountable for their dangerous, unsettling words?

That said, I don’t think ASI will be created by the end of the year. It’s not even clear to me that ASI — whatever that is, exactly — will arrive in the next 5 or 10 years. But this is beside the point. Even if I believe my neighbor’s threat is unfounded, they should still be held accountable for uttering that threat.

The Swiss Alps

Finally, I took off the last two weeks of August to vacation with some relatives in the Swiss Alps. They generously paid for everything, which made the trip possible.

The Swiss Alps are, in my opinion, the closest thing there is to heaven on Earth. The vast pastures, the towering snowcapped mountains, the sprawling meadows and verdant forests, the dreamy ferns and mossy rocks are all glorious notes in a magnificent symphony.2

The way I’ve described the Swiss Alps to friends is this: I grew up in a horizontal world — Frederick, Maryland, 30 minutes away from the puny Appalachian Mountains.

The lucky residents of the Swiss Alps live in a vertical world. As you drive or hike up the switchbacks, you pass sprawling towns almost stacked on top of each other. Enormous mountains, you find, are actually the valleys of mountains above them, which you only discover once you reach the false summit as seen from below. There are, as I like to say, entire worlds atop worlds atop worlds. It’s like walking a circuitous stairway to heaven: at each twist in the journey, you find another, previously hidden staircase ahead of you, and looking back you see mountains and valleys layered on top of mountains and valleys. It’s really something extraordinary, and a reminder of how beautiful our precious oasis in space can be.

Below are some photos and videos from the Swiss towns of Lauterbrunnen, Adelboden, and Leukerbad. The first was visited by J.R.R. Tolkien, and inspired the town of Rivendell. I also visited Zurich at the end of the trip, staying with a good friend of mine (Ged). Zurich is frequently ranked as the best, or one of the best, cities in the world to live in, and Switzerland is consistently ranked as the best country to be born in. Enjoy:

Adelboden below. I hiked a sinuous trail that winds around the huge waterfall. It takes one to a mind-bogglingly huge cirque, i.e., “an amphitheatre-like valley formed by glacial erosion.” You cannot see the cirque from this photo — it is a hidden world. A good example of discovering a world (the cirque) towering above a world (Adelboden) towering above yet another world (the valley below Adelboden).

Lauterbrunnen, next to the accommodations in which I stayed. High up to the left is a sprawling village called Wengen, which looks down on Lauterbrunnen, but is dwarfed by huge mountains that you can’t even see from the Lauterbrunnen valley (itself far above valleys further below).

Wengen, the world over a world over a world, beneath a series of other worlds high above it.

Here’s another picture I took from Wengen when I visited back in 2021. Perhaps you can see why Tolkien was inspired by this place!

Further down the Lauterbrunnen valley:

A video of the Lauterbrunnen valley.

Expansive green pastures in Adelboden (no audio). The snowcapped mountain peaks behind the waterfall are obscured by clouds.

Another video of Lauterbrunnen (no audio):

Another video of Adelboden:

Picture of Leukerbad:

Video of the vast cliffs surrounding Leukerbad. To the right (occluded by the buildings, are snowcapped mountains).

As always:

Thanks for reading and I’ll see you on the other side!

Leftist Conspiracy Theories

On June 12, 2023, the democratic-socialist transhumanist James Hughes and then-graduate student Eli Sennesh posted an essay on Medium in which they argued that the TESCREAL thesis is little more than a leftist conspiracy theory.

This essay was written and posted before Timnit Gebru and I had published even one article on the TESCREAL concept, either together or individually. We hadn’t even given a single interview discussing the idea. Hughes and Sennesh’s entire piece was based on nothing more than a handful of tweets from me and Gebru. This would be like me writing a lengthy, detailed review of Hughes’ book, Citizen Cyborg, based solely on the title and a 200-word blurb.

Naturally, then, their critique completely misrepresents the TESCREAL thesis. Like an LLM, they hallucinated what they thought the TESCREAL thesis might be, and then launched a barrage of rhetorical missiles at the nonexistent target.

That said, the claim that TESCREAL is something of a conspiracy isn’t far off the mark. (This, however, is not what they argued.) It’s worth remembering here that not all conspiracies are conspiracy theories. In a section titled “Some Conspiracies Are Real” of her excellent book Doppelganger, Naomi Klein reminds us that the world has been shaped by “real conspiracies” hatched and executed by the socioeconomic and political elite. She writes:

The poisoned water system in Flint, Michigan, was covered up by state officials year after year. British Petroleum and Halliburton cut corners in the operation of the Deepwater Horizon offshore drilling rig, and the result was the largest accidental oil spill in history in the Gulf of Mexico, with the companies scrambling to cover up the full extent of the damage. Volkswagen ran a conspiracy for years to cover up how much polluting carbon its diesel vehicles were emitting (the cars were programmed to fool the testers). Most consequentially, Exxon and several other oil majors ran a conspiracy to spread doubt and confusion about the reality of climate change for decades, taking a page from the tobacco giants. These are just the barest of examples.

A Dish of Spaghetti Thrown at the Wall

The fact is that the TESCREAL movement is an incestuous tangle of researchers, activists, policymakers, wealthy donors, tech CEOs, think tanks, organizations, institutes, and AI companies all driven by a common eschatological belief system: we must build a Machine God, create a new posthuman species, colonize the universe, and establish a sprawling multigalactic civilization high up on the Kardashev scale.

There is a sense in which this is a conspiracy. The people sharing this belief system all know each other. They’ve all hung out at conferences and private parties. They’ve exchanged ideas on the same email lists. They run the major AI companies, as well as the “AI safety” organizations trying to stop those AI companies.

Even those in the community who appear to dislike each other, such as Eliezer Yudkowsky and Dario Amodei, have extensive personal histories, and despite their disagreements about ASI existential risk, both Yudkowsky and Amodei are trying to realize the very same cosmic “utopia” through the creation of artificial superintelligence (ASI).

To make this more concrete, let’s zoom in on a tiny segment of the social network that comprises the TESCREAL movement, focusing on names like:

• Shane Legg

• Ben Goertzel

• Eliezer Yudkowsky

• Demis Hassabis

• Dario Amodei

• Leopold von Aschenbrenner

• Holden Karnofsky

• William MacAskill

• Nick Bostrom

• Ray Kurzweil

• Elon Musk, and

• Jaan Tallinn

This will get messy, but the messiness is part of the point. I feel confident in saying that you’ll want to read this entire section carefully, maybe twice. It gets more outrageous the further along you read.

***

We begin with Shane Legg, whose 2008 PhD dissertation was titled “Machine Super Intelligence.” For this, he received a $10,000 award from the Canadian Singularity Institute for Artificial Intelligence. That’s the (now-defunct) Canadian branch of Eliezer Yudkowsky’s Singularity Institute, founded in 2000 and later rebranded as the Machine Intelligence Research Institute (MIRI).

From 1999 to 2001, Legg worked for Ben Goertzel’s company WebMind. In the mid-2000s, Goertzel co-edited a book about human-level AI that included a contribution from Yudkowsky, among others. The original title was Real AI, but Goertzel wanted something better. So he asked around for suggestions, and Legg proposed “artificial general intelligence” (AGI, the precursor of ASI). This is how the term “AGI” became part of our shared lexicon, although “ASI” has become more popular among AI hypesters over the past year or so.

Three years after Artificial General Intelligence (2007) was published, Legg cofounded DeepMind with Demis Hassabis, a fellow transhumanist and singularitarian. DeepMind’s explicit goal was the creation of a Machine God. Like many in the ASI race, Legg first became interested in superintelligence after reading one of Ray Kurzweil’s books, though he reports that Yudkowsky got him “obsessed” with ASI. He even exchanged emails with Yudkowsky via the SL4 mailing list, which Yudkowsky created in 2000 as “a refuge for discussion of advanced topics in transhumanism and the Singularity.”

Before founding SingularityNET, Goertzel was an active member of the Extropian movement of the 1990s, along with Yudkowsky, Kurzweil, and Nick Bostrom. At various Singularity Summit events, he shared the stage with these people as well as his former employee Legg and Legg’s DeepMind cofounder, Hassabis (more on this below). The Singularity Summit was an annual conference cofounded by Yudkowsky, Kurzweil, and Peter Thiel.

In 2008, Goertzel was hired as the “Director of Research” at Yudkowsky’s Singularity Institute (now MIRI). The following year, he convinced Jeffrey Epstein to donate $50k to the organization (this money being passed on to another research group, I believe). In fact, Epstein had been financially supporting Goertzel since 2001.1

Yudkowsky was one of the most prominent singularitarians of the early 2000s, alongside Kurzweil. He founded the Singularity Institute (MIRI) to accelerate the techno-rapture and what he frequently calls our “glorious transhumanist future.” He also participated heavily in the Extropian movement, founded Rationalism, and has been highly influential within the Effective Altruist (EA) and longtermist communities. (William MacAskill once described him as a “moral weirdo,” which is among the highest praise one could get from a leading EA.)

The Singularity Institute was initially funded by Thiel ($1,627,000). It also received money from Vitalik Buterin, Jaan Tallinn, OpenPhil (i.e., Open Philanthropy, now called Coefficient Giving), the last of which distributes money from Facebook cofounder Dustin Moskovitz, a hardcore EA-longtermist. OpenPhil has employed Nick Beckstead, Holden Karnofsky, and Luke Muehlhauser, who was previously the Executive Director of Yudkowsky’s MIRI and subsequently became a Board Member at Anthropic.

Beckstead, one of the earliest members of the EA movement,2 cofounded longtermism along with Nick Bostrom, and was a Research Fellow at Bostrom’s Future of Humanity Institute (FHI). After leaving OpenPhil, Beckstead became the CEO of Sam Bankman-Fried’s FTX Future Fund, co-run by Leopold Aschenbrenner, Amitav Balwit, and William MacAskill. (Note that Bankman-Fried was once the most prominent EA-longtermist in the world.3)

The FTX Future Fund had a close relationship with FHI, both sharing the same EA-longtermist ideology. As an article in Asterisk Magazine notes,

for a single, delirious summer, it seemed that every other person at Trajan House [where FHI was located], including FHI staff, was to-ing and fro-ing between Oxford and the Bahamas, where Sam Bankman-Fried was planning a barrage of longtermist philanthropy.

According to the New York Times, “Bankman-Fried and his colleagues had fed at least $500 million into Anthropic” (which we’ll get to in a moment), and the FTX Future Fund offered Nate Soares, a MIRI employee who coauthored a recent book with Yudkowsky, $6k for advice on regranting. Another Bankman-Fried run company, Alameda Research, donated over $113k to MIRI, as well as another $19k to an organization “housed” within MIRI called AI Impacts.

In addition to working with Beckstead at the FTX Future Fund, Aschenbrenner was also an intern at Bostrom’s FHI and its sibling organization at Oxford, the Global Priorities Institute, funded by OpenPhil and cofounded by William MacAskill — an EA-longtermist and transhumanist who was the moral “advisor” of Bankman-Fried.4 (Whoops!) Aschenbrenner subsequently worked at OpenAI as part of the “Superalignment” team, co-led by Ilya Sutskever.

The CEO of OpenAI, Sam Altman, has been pictured partying with Yudkowsky and, like Shane Legg, explicitly credits Yudkowsky with inspiring him to pursue ASI. The New York Times describes him in a profile as being a “product” of the EA (meaning, in this context, longtermist) and Rationalist movements.

In 2024, OpenAI fired Aschenbrenner for sharing sensitive material from OpenAI with Holden Karnofsky, who left OpenPhil in 2024 (as Director of AI Strategy) and joined Anthropic in 2025. In his 2022 book on longtermism, MacAskill credits Karnofsky with having “a particularly profound impact on my broader thinking about longtermism.”

Following in the footsteps of Bankman-Fried, Aschenbrenner pursued the “earn to give” career path invented by EA. He started a hedge fund called Situational Awareness, which imploded this summer, losing $35 billion almost overnight.

This was just days before Aschenbrenner’s wedding to Avital Balwit, the former FTX Future Fund employee who, like her now-husband, previously worked at Bostrom’s FHI and is currently the Chief of Staff to Dario Amodei, Anthropic’s CEO. In attendance at the wedding were Amodei and “more than a dozen Anthropic employees … as well as MacAskill and others from the FTX Future Fund, plus effective altruists and the billionaires who had put their money into Situational Awareness.”

Alongside Beckstead and Karnofsky (who now works at Anthropic), Dario Amodei was one of the earliest members of the EA movement. According to the New York Times,

all of the major AI labs and safety research organizations contain some trace of effective altruism’s influence, and many count believers among their staff members. … No major AI lab embodies the EA ethos as fully as Anthropic. Many of the company’s early hires were effective altruists, and much of its start-up funding came from wealthy EA-affiliated tech executives.

Like Legg, Dario Amodei cites Kurzweil as having piqued his interest in ASI. He’s also shared the stage with Yudkowsky in discussing the promises and perils of ASI, and he spent years living with Karnofsky (they were roommates). Indeed, Karnofsky would later marry Dario’s sister, Daniella, who cofounded Anthropic with her brother after both quit working for OpenAI.

In addition to money from the FTX Future Fund, Anthropic has been funded by OpenPhil (where Karnofsky worked) and Jaan Tallinn (one of the most prolific funders of the TESCREAL movement).

Another noted figure at Anthropic is the utilitarian longtermist Amanda Askell, the former wife of William MacAskill, who was involved in the early EA movement and received her undergraduate degree at Oxford, where FHI and the Global Priorities Institute were based. (MacAskill changed his surname from “Crouch” upon marrying Amanda.)

In 2017, while still an employee at OpenAI, Dario Amodei joined a conference panel on ASI with Nate Soares (of MIRI, also Yudkowsky’s coauthor) and Shane Legg (of the Thiel-funded DeepMind). The conference was hosted by the Future of Life Institute (FLI), which was cofounded by Tallinn and Max Tegmark, the latter of whom has collaborated with Bostrom. Musk and Bostrom were both external advisors at FLI.

Also present at the conference — sponsored by Musk, Tallinn, and OpenPhil — was:

• Musk (who has promoted the work of Bostrom and MacAskill, even calling longtermism “a close match for my philosophy”)

• Eliezer Yudkowsky

• Holden Karnofsky

• Ray Kurzweil

• William MacAskill

• Nick Bostrom

• Max Tegmark

• Jaan Tallinn

• Demis Hassabis

• Sam Altman

• Paul Christiano (who was also roommates with Karnofsky and Amodei)

• Jason Matheny (a former FHI researcher who went on to run IARPA and is now the CEO of RAND Corporation)

• Ilya Sutskever (a noted figure at OpenAI until he quit — Sutskever has parroted Yudkowsky in arguing that ASI could kill everyone on Earth. Recall that Aschenbrenner worked under Sutskever on the “Superalignment” team)

• Helen Toner (a hardcore EA-longtermist who served on OpenAI’s board and voted to fire Altman in 2023)

• Larry Page (a former close friend of Musk’s who personally hired Kurzweil to work at Google)

• Eric Schmidt

• Reid Hoffman

• Vitalik Buterin (recipient of the Thiel Fellowship who donated $665 million to FLI and once participated in an FHI conference)

Going back in time a bit: Seven years before the FLI conference, Hassabis and Legg both gave talks at the 2010 Singularity Summit alongside Goertzel, Yudkowsky, and Kurzweil. (In fact, Legg and Yudkowsky were on the same panel.) Recall that Yudkowsky, Kurzweil, and Thiel founded the summit.

After the summit ended, Hassabis followed Thiel back to his mansion and asked for money to start DeepMind, which Thiel gave him. Hassabis later delivered a talk at the 2011 Winter Intelligence conference hosted by Bostrom’s FHI (image below), and hired Bostrom to be a member of DeepMind’s Ethics & Society research unit.

Taking a closer look at Bostrom, who constitutes a central node in the TESCREAL network: Alongside Yudkowsky, Goertzel, and Kurzweil, he was an early member of the Extropian movement, and he’s become one of the most influential transhumanists today. His 2014 book on the dangers of ASI was blurbed by Musk, and — as noted — he cofounded the longtermist ideology with Beckstead (of FHI, FTX, and OpenPhil).

Bostrom has also been a major figure in Yudkowsky’s Rationalist and MacAskill’s EA movements. He’s coauthored papers with Yudkowsky, and many of his “original” ideas about ASI came from the Rationalist founder. (As Goertzel once wrote, “scratch the surface of Bostrom, find Yudkowsky.”)

Before it was shut down — thanks in part to a racist email from Bostrom that I uncovered in late 2022 — FHI shared office space in Oxford’s Trajan House with the Centre for Effective Altruism, cofounded by MacAskill, who also worked for Bostrom at FHI and, once again, cofounded the Global Priorities Institute that hired Aschenbrenner. In 2015, Musk donated $10 million to FLI, which funneled some of these riches to FHI.

Recall that FHI had close connections with FTX, run by MacAskill’s advisee, Bankman-Fried, who’s now in federal prison. It also had connections with OpenAI. (Altman once described Bostrom’s 2014 book as “the best thing I’ve seen on this topic.”) In 2018, a group of FHI researchers coauthored a paper about AI risk with OpenAI.5 Notable names on the paper include Helen Toner, Dario Amodei (still at OpenAI, along with his sister, Karnofsky’s wife), and Carrick Flynn.

As it happens, Flynn was a former FHI researcher who ran for congress several years ago in Oregon. His campaign boasted of more than $10 million from Bankman-Fried, which “is considered an astronomical figure for a House Democratic primary.” His campaign manager? None other than Balwit, the former employee under Beckstead at the FTX Future Fund, who now works for Dario Amodei and is married to Aschenbrenner.

A Conspiratorial Afterimage

Believe me when I tell you that this is a tiny, tiny glimpse at how incestuous the ASI race — driven by the TESCREAL movement — is. An exhaustive account would require, I’d say, 100 pages of dense reading like that above.

If one were to compile such a document, you’d find the same people, think tanks, companies, etc. popping up over and over again. As mentioned earlier, these people all know each other. They’re all part of the same fringe eschatological movement (though that movement is hugely powerful in Silicon Valley, and hence the world more generally) with the shared goal of building a Digital Deity and conquering the cosmos as radically transformed digital space brains. Even a cursory glance at this network leaves a conspiratorial afterimage, like the floating dots of colors you see after looking at a bright object and then closing your eyes.

Indeed, it’s worth adding briefly that many notable media figures have connections to this movement, including Ezra Klein, Sam Harris, and Kelsey Piper. The last of these knew Bankman-Fried, is friends with Yudkowsky, and writes for the Vox vertical Future Perfect, which was funded by FTX. Even more striking is the fact that Ross Douthat, formerly of the New York Times and now a correspondent at CBS’s 60 Minutes, officiated Aschenbrenner’s wedding. (Side note: the New York Post reports that Aschenbrenner promised “his future wife he would someday buy her not a yacht or a mansion, but a galaxy.” Lolz.)

Here’s how Wikipedia defines a “conspiracy”:

[A] plan or agreement between people … for an unlawful or harmful purpose, such as murder, treason, or corruption, especially with a political motivation, while keeping their agreement secret from the public or from other people affected by it. In a political sense, conspiracy refers to a group of people united in the goal of subverting established political power structures. This can take the form of usurping or altering them, or even continually illegally profiteering from certain activities in a way that weakens the establishment with help from various political authorities.

I don’t think the TESCREAL movement, which admits of many minor variations, is cohesive enough to justify calling it a “conspiracy” without significant qualifications. But it is conspiracy-like. The difference between a real conspiracy and a conspiracy theory is, of course, epistemological: it depends on the totality of the available evidence. As Naomi Klein implies, there is ample evidence to justify describing Big Oil’s efforts to mislead the public about climate change as a conspiracy. But there isn’t any good evidence (!) to support the claim that commercial airplanes are spewing biological or chemical agents into the atmosphere, or that 9/11 was an inside job.

If one seriously examines the cluster-fuck of connections within the TESCREAL movement, and between the TESCREAL movement and the leading AI companies, it becomes clear that calling it “conspiracy-like” is epistemologically warranted. Or so I would argue.

Subscribe now

But what do you think? What am I wrong about? As always:

Thanks for reading and I’ll see you on the other side!

Since I’m technically on vacation right now (until September 1), I thought I’d publish a relatively short piece on a phrase I hear a lot these days: “the danger of the race to build ASI” (artificial superintelligence). It’s worth noting, I think, that this phrase is ambiguous. It can mean two radically different things.

On the one hand, people in the field of AI safety — many of whom are “doomers”1 — believe the ASI race is dangerous because it will result in ASI. Specifically, they believe it will likely result in misaligned ASI unless significant steps are taken to ensure that ASI is “value-aligned” (by which they typically mean “aligned with the TESCREAL worldview”). Many of these people will add that ASI is imminent, and hence that everyone on Earth will die in the very near future.

On the other hand, when people like me talk about “the dangers of the ASI race,” we typically mean that trying to build ASI itself poses significant threats to individuals, democratic societies, and perhaps even humanity itself. Many of us don’t believe that ASI is imminent — if it’s even possible to build, assuming the term is meaningful in the first place. But this is beside the point. The fact is that rushing toward what the AI companies call “ASI” has inflicted, and will continue to inflict, serious harms on society and the individuals who comprise it.

Hence, worrying about the dangers of the ASI race doesn’t commit one to a belief that ASI will arrive anytime soon, if ever. By analogy, you could imagine multiple companies racing to build a perpetual motion machine (PMM). They pour huge amounts of money into their respective projects, polluting the environment, exacerbating climate change, destroying communities, etc. in the process. Despite the fact that PMMs are physically impossible, according to the laws of thermodynamics, one could still talk about “the dangers of the race to PMM” — an imaginary end-goal that these companies will never reach.

The same goes for the ASI race. I think this is important to note because referring to “the dangers of the ASI race” could be misunderstood as suggesting that one believes ASI is possible and imminent. One interpretation of the critiques of scholars like Emily Bender, Alex Hanna, Timnit Gebru, etc. is that they are deeply concerned about the ASI race — that is, it’s deleterious consequences — despite them believing (as I understand their views) that ASI isn’t even a coherent concept.

Whenever you hear someone talking about the ASI race being dangerous, it’s worth taking a minute to decipher what they mean. Are they referring to the race itself, or the anticipated outcome of this race?

AI Ethics vs. AI Safety

This ties into another issue worth mentioning, I think. It concerns the distinction between AI ethics and AI safety. One often hears this distinction elucidated as follows: AI ethics focuses on near-term risks whereas AI safety focuses on long-term risks.2

But this is misleading. For example, AI ethicists worry about the environmental consequences of current AI systems — seen by the AI companies as the stepping-stones to ASI. (Hence, to build ASI, we must first build systems like ChatGPT-5, Claude Mythos, etc.) AI has a nontrivial carbon footprint, and indeed the CO2 emissions of companies like Google and Microsoft have risen — despite promises to be carbon-neutral or carbon-negative by 2030 — due almost entirely to AI.

But, of course, climate change is a long-term risk. The CO2 injected into the atmosphere right now will linger for millennia, affecting the livability of Earth for the next 10,000 years — a longer period of time than civilization has so far existed.3

On the flip side, many AI safetyists and doomers believe that ASI could arrive in the next few years, and that, since we don’t yet know how to “align” it with our (i.e., the TESCREAL worldview’s) values, it will annihilate humanity by default. This is clearly a near-term risk, as it would mean the untimely death of you and me, our children and parents, and everyone else alive a few years from now.

A better way of understanding the difference between AI ethics and AI safety is in terms of the distinction made above. Those in the first camp are mostly concerned with the externalities of attempting to build ASI, while those in the second tend to ignore or minimize the inimical effects of the race itself, focusing instead on the “existential” stakes of actually building a superintelligent machine.

Herein lies the most salient difference between these two fields.

My Current View

What are my views on the matter? I mentioned above that my primary concern is the harms caused by trying to build ASI. Such harms include:

• exploiting workers in the Global South,

• stealing our intellectual property,

• exacerbating climate change,

• sucking up massive amounts of energy,

• flooding the Internet with slop, disinformation, and deepfakes,

• inducing episodes of psychosis in vulnerable populations,

• perpetuating harmful stereotypes due to algorithmic bias,

• enabling unprecedented levels of mass surveillance,

• working with the US military,

• and so on.

I see no reason for believing that ASI is imminent, and indeed the TESCREAL pro-extinctionist Sam Altman recently acknowledged that the hype around ASI has been overblown. Altman, of course, has been one of the biggest hypesters out there, even arguing last year that the Singularity has already started.

I’m not even sure that the term “ASI” is meaningful, as no one can agree on a definition and the concept of intelligence itself is poorly defined. (To a great extent, our contemporary notion of intelligence has been influenced by the eugenics movement, which has morphed into the TESCREAL movement. Hence, Gebru and I characterize TESCREALism as the most recent iteration of the “eternal return of eugenics.”)

That said, it also seems clear to me — I think — that if the AI companies were to build a system that could outmaneuver us in every important respect, solve complex problems much faster than we can, set and modify its own goals without human intervention, and act as autonomous agents in the world, then we’re in very big trouble. I don’t see any good reason for thinking that our species would survive for very long competing against an artificial system of this sort. You can call that “ASI,” but you don’t need to, to grasp the danger.

Hence, if the ASI race succeeds in building an “ASI,” the outcome would likely be terrible. (Even the “good” outcomes would be terrible, as these would also inevitably lead to the extermination of our species.) But I don’t think the ASI race will “succeed,” at least not anytime soon, and thus when I talk about “the dangers of the ASI race,” I mean “the dangers of trying to build this thing called ‘ASI,’ whatever that is exactly.”

But what do you think? What am I missing? Is this distinction useful? (It is in my own mind!) As always:

Thanks for reading and I’ll see you on the other side!

Summary1:

A woman named Cami Clark, who hung out with EA-longtermists and married one of the most powerful EA-longtermists in the world (Dario Amodei), exchanged emails with Jeffrey Epstein after his 2008 conviction in hopes of getting him to fund her porn company, whose tagline was: “intellectually promiscuous.”

Another leading EA-longtermist, Leopold Aschenbrenner, who worked for Bankman-Fried’s FTX Future Fund and married a former colleague at the Future Fund, Avital Balwit — who now works for Clark’s husband, Amodei — started a hedge fund that invested heavily in Anthropic and recently lost $35 billion of its $45 billion AUM.

Lol.

Dario Amodei’s Wife Tried to Get Jeffrey Epstein to Fund Her “Luxury Porn” Company

Last February, I wrote a Truthdig article about Jeffrey Epstein’s extensive connections to the transhumanist community. Epstein himself was a transhumanist and eugenicist (but I repeat myself) who wanted to have his head and penis cryogenically frozen and aimed to create a “super-race of humans with his DNA” by impregnating up to 20 women at a time on his Zorro Ranch.

He was in contact with Bryan Johnson, and appears to have met Aubry de Grey (the longevity researcher) over Skype. He was also pals with Peter Thiel and Peter Diamandis, and exchanged emails with Elon Musk, who practically begged Epstein to let him visit the pedophile and sex trafficker’s home-base in the Virgin Islands.

Epstein was also long-time buddies with Ben Goertzel, who congratulated Epstein when he emerged from prison in 2009, convinced Epstein to transfer $50,000 to Eliezer Yudkowsky’s Singularity Institute (now the Machine Intelligence Research Institute), and wrote the following to Epstein after sending him birthday wishes:

Now if you want to have 100000 more birthdays you should fund my AI project so we can create a thinking machine that will solve the problem of mortality!!!

These are just a few examples of Epstein’s connections to the TESCREAL movement.

***

It turns out that Epstein was also friendly with Cami Clark, the wife of Anthropic CEO Dario Amodei. Previously, Clark had dated Eric Schmidt, the former Google CEO who claimed we should build ASI (artificial superintelligence) despite the environmental costs. She married Amodei in 2022 (having been married once before to an architect who was 64 years old while she was just 20).

As Forbes reports:

• Her previous attempted businesses include founding a self-described “free luxury porn” company that catered to women and a “social dieting app.”

• Clark met Epstein in March 2011 through John Brockman, a literary agent who knew Epstein, with emails in the Epstein files showing Brockman telling Epstein he “should connect for dinner with my girls,” referring to Clark and her business partner Michelle Capocefalo. [The full quote is: “You should connect for dinner with my girls — Cami and Michelle. They’re in LA raising money for porn movie aimed at women’s market.”]

• “We would love to have dinner with you this evening,” Cami said in an email responding to Brockman’s introduction, then sending an email the next day that passed along her company’s funding requirements and exclaimed, “It was FANTASTIC meeting you last night!”

• Clark then asked Epstein in a March 2012 email exchange about whether he’d be interested in investing in her and a colleague’s businesses, though there’s no indication in the Epstein files that the financier made any investment.

The porn company was apparently called Eddice. It “described itself as a ‘revolutionary porn company’” that emphasized “sex positivity in a male-oriented porn industry, and carried the tagline: ‘intellectually promiscuous.’”

All of this happened after Epstein pled guilty to soliciting prostitution from a minor and spent 13 months in prison — out of the 18 months he was sentenced to in a “sweetheart deal.” Upon being released, he almost immediately began raping an 18-year-old woman who had “recently arrived from Uzbekistan.”

As with Noam Chomsky, who became close friends with Epstein, Clark appears not to have cared that Epstein was a registered sex offender.

Forbes continues:

After their initial introduction through Brockman, Clark sent Epstein the script and treatment for her company’s first films a few days later in March 2011, noting she thought Epstein “and the ladies” might enjoy them and saying the content was “a little nsfw.” Clark’s March 2012 exchange with Epstein is the lengthiest conversation between them that appears in the Epstein files, with Epstein initially asking, “i’m sorry, who is this,” after Clark reached out and asked to put Epstein in touch with another female entrepreneur launching a lingerie line. When Clark reminded him that she has the “free luxury porn company” and asked if that “ring[s] a bell,” Epstein responded, “yes, a loud gong.” (“I was going to say, you would be the first person ever that didn’t remember us :),” Clark responded.) … Epstein then asked what happened to Clark’s business plans with the porn company and she said they were moving “full speed ahead,” pitching Epstein on investing in her company’s foray into video. “Can’t do TV sex,” Epstein responded, to which Clark said, “I didn’t think so :).”

Forbes adds that “other emails in the Epstein files show Clark keeping the financier on her email list — inviting him to a housewarming party in July 2012 — and that Epstein connected with her on LinkedIn a year later in 2013.”

***

Clark has almost no online footprint. She keeps a low profile, but appears to be influential behind the scenes, even playing a role in getting Anthropic early funding. As the Wall Street Journal, which broke the story, observes:

When Indian Prime Minister Narendra Modi invited AI leaders to a meeting in New Delhi earlier this year, security protocols allowed each executive to bring one additional person with them. Most brought colleagues, but Anthropic CEO Dario Amodei brought his wife, Cami Clark.

Clark doesn’t work at Anthropic, but she is often seen sitting in the front row while Amodei talks at events such as Davos or can be found chatting up investors at gatherings such as the Allen & Co. conference in Sun Valley.

She acts as a sounding board and strategic adviser for Amodei, according to people close to the company. She also brought Anthropic a key early investor, former Google CEO Eric Schmidt — whom she had dated — as it was getting off the ground in early 2021, some of the people said.

WSJ also notes that

Clark spent time at the group house in San Francisco where Amodei, his friend Holden Karnofsky and his sister, Daniela — who would later marry Karnofsky — lived with other key figures from the AI safety and Effective Altruism worlds. The movement, centered largely in Silicon Valley, aims to use data and reason to determine how to do the most good possible.

And:

Clark has recently told people involved in politics that the company sees its mission as protecting America and isn’t as “woke” as its detractors might think it is, according to people familiar with the matter.

At the Sun Valley conference in July, where she spent much of the week by her husband’s side, she had lunch with Ivanka Trump, a friend, and chatted with Jared Kushner, whom Amodei had approached earlier in the year to invest, people familiar with the matter said.

She strikes me as a hyper-ambitious person with low moral standards, which of course describes many power-hungry vultures in Silicon Valley — including her husband. I wonder who else in the TESCREAL/ASI race has had connections with Epstein that haven’t yet been reported.

Womp Womp: Situational Awareness Loses $35 Billion

Leopold von Aschenbrenner — a German-born investor and AI safety researcher whose name I jokingly add the aristocratic “von” to — is an Effective Altruist (EA) and longtermist who gained a bit of Silicon Valley fame for his wildly hyperbolic 2024 essay, “Situational Awareness.” In it, he hyperventilates about ASI being imminent and argues that it’s very important for the US to win the ASI race against China.

Here’s what I write about his essay in my forthcoming book on the TESCREAL ideologies:

Aschenbrenner’s “impressive” CV includes a stint at Sam Bankman-Fried’s cryptocurrency exchange platform, FTX. Specifically, he worked for the FTX Future Fund, run by Nick Beckstead — the longtermist who argued that we should prioritize saving the lives of people in rich countries. The Future Fund also gave $500 million in funding to Anthropic.

While still a teenager, he wrote a paper titled “Existential Risk and Growth,” and cofounded the Effective Altruism club at Columbia University when he was a student there. He spent time at the Future of Humanity Institute (before it was shut down due in part to a racist email from it’s founder that I uncovered in late 2022), as well as Will MacAskill’s Global Priorities Institute. He later joined OpenAI’s “Superalignment” team, but left after OpenAI accused him of sharing confidential documents (which von Aschenbrenner disputes).

Following his viral-in-the-Valley 2024 essay, he founded a hedge fund of the same name: Situational Awareness, which he describes as “an investment firm focused on AGI.” At its peak, Situational Awareness had $45 billion assets under management (AUM), but last month it lost a catastrophically huge $35 billion. That still leaves an eye-watering $10 billion behind, but the massive drop in assets reminded many of the ignominious collapse of Aschenbrenner’s former employer: Bankman-Fried.

As CNBC reports,

critics pointed out that Aschenbrenner had no experience running money prior to launching his fund in July 2024, calling him more lucky than smart. Some noted that his early work experience was at the doomed crypto firm FTX, where he helped now-disgraced founder Sam Bankman-Fried run a charity out of a Bahamas penthouse.

“A lot of people saw this blow-up as a matter of not if, but when,” said Jerry Diao, who runs a Wall Street coaching firm. “Maybe his views on AI are correct in the long run, but in the public markets, you have to be prepared for the short-term.”

The same article opens with the rather sardonic line: “Two years ago, Leopold Aschenbrenner argued he was one of few people in the world who saw the future clearly.” Now, von Aschenbrenner has made himself the butt of jokes and thrust EA into the spotlight once again. (Aschenbrenner was essentially earning to give, the very same career path that led Bankman-Fried into crypto.)

As my friend writes in a quite amusing article, “Son of Sam: Aschenbrenner was SBF 2.0”:

An infamous Effective Altruist thought experiment is “the coin flip to end the world.” Sam Bankman-Fried interpreted this as an injunction to take catastrophic risk if the average “expected value” of winning a long bet was even slightly positive.

That high-risk mindset also seems to have ruled at Situational Awareness, a huge, AI-focused hedge fund that exploded in blood and guts last week. It was run by a deeply involved Effective Altruist named Leo Aschenbrenner.

Aschenbrenner’s downfall very closely tracks that of Sam Bankman-Fried. Specifically in that they were both young, loved using very high leverage in volatile markets, believed trend-lines only go up, and apparently thought risk management is for cowards.

Morris continues:

Aschenbrenner still got caught by the exact same ratchet that killed SBF.

Situational Awareness LP grew incredibly fast after 2024, to a peak of $45 billion in AUM, by using high leverage to bet on the continued rapid growth of AI. Aschenbrenner had kept making the same bet for years, at larger and larger size, to the point where his leverage may have been propelling the broader market.

Then July was choppy for AI stocks — incredibly, even $Nvidia was flat - and that was all it took to destroy Situational Awareness. …

[T]he linear, predictionist modelling that rules EA, alongside the idea that its adherents are uniquely genetically gifted, lead believers to take on extreme risk and, inevitably, to blowups like Aschenbrenner’s.

He further notes that, almost comically,

Aschenbrenner’s unluckiest investors lost something like 110% of the current mark-to-market value of their assets.

That means Leo Aschenbrenner was far, far worse for many investors than Sam Bankman-Fried was for FTX depositors.

***

Of note is that von Aschenbrenner’s wedding to Avital Balwit happened just weeks after the $35 billion loss. Balwit is Dario Amodei’s chief of staff at Anthropic and a former employe at Bankman-Fried’s FTX Future Fund — just like her husband was. Note also that Situational Awareness had a multi-billion-dollar investment in Anthropic. As Morris observes, “the largest remaining asset in Situational Awareness LP portfolio is reportedly a large stake of illiquid Anthropic equity.” (This community is so incestuous!)

Balwit, I should add, wrote a long response back in 2021 to a couple of articles I published critiquing longtermism — an ideology that both she and her husband obviously subscribe to. Her critique of my critique is pretty abysmal, spurring me to write a response in which I dissect each paragraph, showing how far her criticisms miss the mark. I had some personal communication with her because I sent her my response prior to publication for comments. (A courtesy I always extended to EAs and longtermists, though they never reciprocated.)

Anyways, the whole von Aschenbrenner debacle is rather hilarious. As I wrote years ago, EA is the gift that keeps giving to critics.

As always:

Thanks for reading and I’ll see you on the other side!

PS. In case you need a chuckle, I saw this on Facebook the other day. :-)

CORRECTION: In the original post, I accidentally wrote “$45 billion” on a few occasions instead of “$35 billion,” which is how much Situational Awareness lost. My apology for the error!

The following piece is far longer than usual. It will be most relevant to those with a strong interest in the TESCREAL thesis. The footnotes, in particular, contain a wealth of information about how I’ve outlined and elucidated this thesis over the years. A shorter, more accessible article will be published tomorrow. Thanks so much, friends!

tl;dr

1. Connor Leahy offers an intellectual history of the Silicon Valley worldview that is strikingly similar to the history outlined by Timnit Gebru and myself.

2. He highlights the TESREAL (no “C”) ideologies as having played an integral role in launching the race to build ASI (artificial superintelligence), whereas we highlight the TESCREAL ideologies.

3. In copious footnotes, I show that many of Leahy’s claims are nearly identical to claims I’ve made in articles and interviews over the past many years.

4. This cannot be a coincidence, since it’s impossible to work in the space of AI and not have come across my work (solo and with Gebru) on the TESCREAL bundle.

Very, very few people have told this story in any good sense. I would love for more people to tell this story ‘cause it’s [a] fucking crazy story, man. Oh man. So the story as we have it today with, like, you know, this, like, ChatGPT and AGI and ASI and AI risk. All of this stuff descends from this one weird offshoot 1980s group of futurists called the Extropians.

— Connor Leahy, November 6, 2025.

Connor Leahy is the cofounder of Conjecture, an AI safety organization at which he was CEO until this year, and he cofounded a nonprofit AI research company called EleutherAI. He is currently the US Director at ControlAI, which is seeking $50 million in annual funding. According to Wikipedia, “he has testified before the House of Commons of Canada and participated in the inaugural AI Safety Summit in 2023.” He has appeared multiple times on CNN, as well as CBS, ABC News (Australia), Al Jazeera English, BBC, and countless popular podcasts, stacking up millions of views.

He mentions in many interviews that he speaks to policymakers in DC, where he’s based, about AI existential risk. If you read or write about AI, there’s no way you’ve missed him. He has become one of the leading voices warning about the annihilatory dangers of misaligned ASI (artificial superintelligence).

Recently, he gave an interview on the Onezero podcast hosted by Robert Wright, whose work I admire. It’s titled “The Ideology Behind AI,” which happens to be a primary focus of my research over the past many years. I found this interview to be deeply problematic, and am inclined to describe Leahy’s conduct as unethical.

But first, it will be useful to provide an overview of Leahy’s basic argument. Take note, if you’d like, of the copious footnotes pointing to articles, interviews, and tweets in which I have made virtually identical claims. My apologies for this overview being rather long (~2,000 words), but I believe this is necessary to make my point. Here goes:

Leahy’s Analysis

Note that I have slightly edited the quotes below to improve readability (e.g., removing words such as “like” and “you know”). The meaning remains intact.

“There’s a lot of really fascinating history” with respect to the “milieu” or “scene” out of which the ASI race emerged, Leahy says. The goal of creating God-like machines “didn’t come out of a vacuum. This is not the ruthless market decided [sic] this is the optimal thing to go for. These are ideas that actually grew out of a very specific ideological lineage that I would say really started with what is called the Extropians.”1

Extropianism, Leahy says, is a “particularly intense form of what’s called transhumanism more broadly. Transhumanism is a very, very broad set of philosophies that in various ways are about belief that humanity should be improved or transcended in various ways,” in particular, through technology that enables us to reengineer humanity.2

He says that “San Francisco as a place is, and Silicon Valley is, a really weird place.3 And it really has, I would say, three main ideological strengths that kind of all mix and mutate in this stew. And these three are the transhumanist, techno-utopian kind of visions4 and also coming from golden-age sci-fi, and stuff like that, which we’re talking about. The other one is just cutthroat capitalism — libertarianism. And the third is true new age communist, you know, crazy hippie ideology. All of these are core to the soul of Silicon Valley.5 And if you neglect any one of them, you won’t understand what’s actually going on. So, all of these cultures of all these [AI] companies … should be understand [sic] as being influenced by these three large ideological clusters to varying degrees.”6

“So,” he continues, “this is where I think the origin of this culture kind of comes from. And then the modern version of what you now see in many of these, like, AGI religions or cults almost7 is coming from the Extropians,8 who are a particularly intense form of transhumanism that kind of emerged in the 80s by a guy named Max More — I think was the one who first coined the term.9 Extropianism was all about exponential growth.10 It was this idea that we could get exponential amounts of energy and growth and intelligence, and improve everything at unbelievable levels …”

He adds that Extropianism “was one of the first ones to really seriously grapple with some very intense ideas such as whole-brain uploading, immortality, and AGI … and superintelligence as concepts.11 They were some of the first people to really think seriously about some of these ideas in the 80s and the 90s.”

Leahy describes how Eliezer Yudkowsky befriended Max More and his wife, Natasha Vita-More, and how he “was someone who was very early on the Extropian mailing list, and so on, talking about these ideas of superintelligence. They didn’t exactly call it that back then, but that’s what they were talking about: this idea of, if you had this recursively self-improving AI that could improve itself, you could … build these incredibly powerful superintelligent beings that could cure all diseases, could solve all problems, could turn the whole universe into … nanobots, or whatever — like, really intense sci-fi stuff.”12

Leahy then notes that the SL4 mailing list grew out of the Extropians mailing list and Yudkowsky’s “interest in this topic in the late 90s and early 2000s.13 He actually splinters off from Extropians and creates” this alternative mailing list, “which is where we see some of the first overlap with some names you might otherwise recognize [from the AI world]. SL4 was a mailing list for talking about these most intense kind of — even more intense than Extropianism. Like, the most intense forms of future sci-fi speculation.” Names associated with the mailing list include Shane Legg, cofounder of DeepMind, and Nick Bostrom.14 Wright then mentions that Shane Legg coined the term “AGI” (artificial general intelligence, the precursor to ASI, though it’s sometimes used as shorthand for ASI), to which Leahy says, “I think he did. Yes, if I remember correctly.”15

Leahy observes that Yudkowsky “founded an organization called the Singularity Institute” in 2000.16 “So, very ahead of his time.” The initial “goal was to build AI as fast as possible so that we could have AI that can take over the world and fix all the problems. … And so, at this time, this is what really would nowadays be called accelerationism.17 So, you know, the e/accs [effective accelerationists] are little, little babies compared to” Yudkowsky — “he was 20 years ahead of them,”18 although “there’s a broader category of accelerationism that I think is not specific to [him].” It has a connection to Nick Land.19

Leahy then adds: “Fun fact, one of the first people to give [Yudkowsky] money was Peter Thiel.”20 But Yudkowsky, in contrast to Thiel, later came to believe that “if you build something that smart, by default, this does not go well. And so he pivots entirely to thinking about, ‘How do you build safe AI?’ How do you build safe — back then it’s called ‘Friendly AI,’ but what we would call ‘safe’ or ‘controllable’ or ‘aligned.’” Consequently, Yudkowsky “started developing this ideology that he would call Rationalism.”21

He continues: “So, you said this kind of earlier which, like, oh there’s all these things going on that we just didn’t know about and it wasn’t like they were hiding it. Well you would be wrong: they were in fact hiding it. So, this was a thing in the early days of this movement — very early on there was a strong part of the ideology is that don’t tell people — don’t let the stupid unwashed masses know about this. Don’t let the government know about this because they’re, you know, they’re stupid. They don’t understand it. We the smart elites who are super-smart, we’re going to solve the problem. You know, we’re going to solve everything. You know, I just have to make sure that all these dumb, stupid, you know, normal people don’t get in the way.”22

Wright brings up the infighting in these groups, to which Leahy says: “this feels like it happens every six months — there’s a lot of infighting inside of these milieus.” He adds that one reason he’s “broken a lot with these group [sic] of people is because I think it is extremely bad to hide this kind of information from people in the government. I think people are allowed to have insight and make decision-making [sic] on this thing [i.e., building a Digital God]. Doing this unilaterally is deeply immoral.”23

Moving on with the story, Leahy notes that in the “second half of the 2000s, a couple really interesting things starts [sic] happening. One thing is, is that DeepMind is founded. DeepMind, including seed capital from Peter Thiel,” thanks to Yudkowsky introducing “Peter Thiel to Demis [Hassabis] and Shane [Legg] to get some other initiative.”24 Although not “all of the money came from Peter Thiel or this one conversation [after the 2010 Singularity Summit], but it still was a pivotal moment and shows this overlap between the DeepMind and kind of the [Yudkowsky] and, like, this race dynamic thing.” The goal was to get all the smartest people “into one company and keep the government and everyone else away, then we can do it right. Then we can do it properly because then we will have control.”

“So,” Leahy says, “I think we’re starting to assemble all the ideological building blocks that lead to the rickety edifice that we’re at now, where there’s one block missing, which is kind of Oxford University utilitarianism — Peter Singer and [William] MacAskill, which led to what is called Effective Altruism.”25 He notes that “in a sense there’s kind of a bait-and-switch, where you read the head on the tin, what EA says, and you say, ‘Well, that seems very reasonable,’ but then if you open the tin you’re suddenly going to see a lot of crazy stuff.”26

There is only one possible thing Leahy could be referring to here, and that’s longtermism. Some longtermists themselves say that the ideology leads to “crazy town.” As footnote 24 notes, longtermism was the primary focus of EA from its inception.

He continues: “You’re [sic] suddenly see your Anthropics, and your whatever — your people saying that actually replacing humans with AI is good, maybe, or that we have to race to ASI or like many, many crazy things.”27 What happened here? “What happened basically is that this pretty non-objectionable version of EA28 merged through this shared link of utilitarianism with Rationalism and this libertarianism, Bay Area Silicon Valley culture, to create kind of the new mutant ideology. And this grew out to a large degree from — so, one of the largest and most important early EA organizations was called GiveWell.”

What’s that? GiveWell “was an organization that evaluated charities for effectiveness and would help you make better donations, which I think is a very reasonable thing to do. It was founded by a previous hedge fund manager named Holden Karnofsky. And Holden later then got to know one of the Facebook founders, Dustin Moskovitz. Dustin Moskovitz decided to donate a very, very large amount of money to a foundation that Holden Karnofsky would lead, and this became what is called Open Philanthropy.”29

That organization’s money has “gone to OpenAI, to Anthropic, to METR,”30 along with lots of AI safety groups. “What is important to understand about Open Philanthropy is that it is the beating heart of the financial system of EA — is that the engine that kind of fed Effective Altruism to a large degree in Silicon Valley specifically [i.e., the longtermist movement] came through Dustin Moskovitz’s money being funneled through Open Philanthropy. It’s hard to overstate how big of an influence they were.”31 Even more, Karnofsky married Dario Amodei’s sister, Daniella, and “in fact, Dario and Holden were roommates for over 10 years.”32

A key point here is that, Leahy says, “what you’ll see is that the same names keep popping up quite a bit. You can kind of pick and choose almost anyone of the, you know, here and there will be a connection. There will be a connection to OpenPhil or to MIRI. … DeepMind. These people all know each other.33 And this is why I think it’s so fascinating, is that there was an AGI superintelligence scene long before the technology existed. Long before it existed, and it overlapped very heavily with these other adjacent ideologies like Rationalism and Effective Altruism, and so on. And this world is extremely incestuous — extremely incestuous.”34

Leahy notes once again that these are basically cults and, after Wright asks about the Venn diagram of all the ideological communities, Leahy describes it as “an extended cluster that is located primarily in Berkeley and San Francisco.” He adds: “There’s just an extended scene. It’s not one ideology. It’s a scene.35 Some of it is EA; some of it is Rationalism; some of it is just AI company employees.”

That said, it’s not the case that “everyone in Silicon Valley or at these AI companies is deeply part of this culture. But you’d be surprised how many are. You’d be surprised how much this culture has been very influential on the thoughts of people. Even the accelerationists, who … love to present themselves as the anti-EAs [but] they’re part of the same culture. They’re the same. … It’s just like there’s two bits of difference, but it’s like … almost everything is the same. It still comes with this techno-utopian vision, this vision of superintelligence, this openness to a lot of alternative strange things,” etc.36 Leahy even adds that there is a lot of tension between some of these groups, drawing this analogy: “There’s the saying, ‘the only thing a leftist hates more than a rightwinger is another leftist,’ right?”37

At one point, Leahy declares to Wright that “the extended cluster of like all of these adjacent ideologies and beliefs38 doesn’t really have a name.”39 Wright also says during the conversation, rather humorously: “You know, I gotta say for people who [are] younger than me with great literary ambitions there is a great intellectual history to be done about all this, right?,” to which Leahy replies: “Oh yes, oh my god, like, there is. That will be unlike other intellectual history you know.”

“TESREAL” Versus “TESCREAL”

That’s the gist of Leahy’s analysis. I encourage you to listen to the entire interview yourself, beginning around 14:00, as there are many details that I don’t have space to mention here.

What do you think? Does Leahy’s analysis, um, sound familiar? He weaves together an intellectual history in which the main characters are every letter in the TESCREAL acronym, presented in roughly that order, except for cosmism.40 That is, Leahy highlights:

• Transhumanism.

• Extropianism.

• Singularitarianism. (The idea that we should bring about the Singularity by triggering an “intelligence explosion,” in I. J. Good’s phraseology, via recursive self-improvement.41)

• Rationalism.

• Effective Altruism (EA).

• Longtermism. (As noted, this is the version of EA that’s gained a foothold within the Valley. When people talk about EA in SV, especially with respect to the ASI race, they’re almost always referring to longtermism.)

He adds that e/acc is yet another ingredient in the alphabet soup of ideologies, just as Gebru and I do, and as I do in numerous solo articles and interviews.

His analysis centers around the TESREAL (no “C”) ideologies, plus “e/acc.” In exactly the same way, my analysis with Gebru centers around the TESCREAL ideologies, plus a “variant” called “e/acc” — which I have elsewhere characterized as a major camp within the TESCREAL movement, the other being the doomers.42

Furthermore, Leahy refers to this as a “cluster” of ideologies, just as I’ve called it a “constellation” and, back in 2023, a “cluster of ideologies.” In our coauthored paper, Gebru and I refer to TESCREAL as a “bundle.” Leahy says that these ideologies constitute “an extended scene,” whereas Gebru and I describe this cluster as a “movement” (basically synonymous with “scene” in this context). Leahy also asserts, as noted above:

I’m not saying by any means everyone in Silicon Valley or at these AI companies is deeply part of this culture. But you’d be surprised how many are — you’d be surprised how much this culture has been very influential on the thoughts of people.

Gebru and I write:

It is important to note that not everyone associated with ideologies in this bundle believes in the totality of the dominant views in this bundle, and some people may even object to being bundled in this manner. Many people working on AGI may be unaware of their proximity to TESCREAL views and communities. Our argument is that the TESCREAList ideologies drive the AGI race even though not everyone associated with the goal of building AGI subscribes to these worldviews.

If you noticed the many footnotes I added to the previous section,43 you’ll see that nearly every important point that Leahy makes has been made previously by me (solo or with Gebru), sometimes using rather similar language. This isn’t just about the details. It’s about how those details hang together to form a panoramic narrative of intellectual history.

Indeed, many people have said things that are similar to particular claims found in my work (solo and with Gebru). But Gebru and I were the first to assemble the heap of these claims, along with many additional new claims, into a comprehensive picture of the origins of the AGI race.

Leahy’s TESREAL (no “C”) analysis is little more than my analysis with Gebru, minus our emphasis on the role that modern cosmism played in inspiring the ASI race. (The term “AGI” was popularized by a transhumanist, Extropian, singularitarian who founded modern cosmism. He also played an integral part in getting the field of ASI research off the ground. A deeper analysis would reveal that cosmism is an essential part of this intellectual history.)

It’s not that Leahy’s account is somewhat similar to ours/mine. The problem is that it’s virtually identical in every important sense.

What Others Think

Indeed, the reason I became aware of his interview with Wright was because a friend of mine, a lecturer (we’d say “assistant professor” in the States) who’s published extensively on transhumanism, sent it to me along with the message:

He’s saying a lot of the stuff you and I talk about all of a sudden. Being clear about the link between transhumanism and the AI industry. Basically talks about them being a cult, doesn’t use the term TESCREAL. Feels like he’s been reading your articles. But doesn’t attribute. … He goes through the whole TESCREAL thing without ever saying TESCREAL. … I sent you that video cos I really thought he’s saying all your stuff without credit. It almost was weird him not mentioning [sic] the term TESCREAL given his thesis.

(Note that the quote above and all those below have been reproduced with permission.)

After watching the interview myself, I was quite shocked by the significant and extensive similarities. So, I sent it to numerous friends and colleagues, asking for their opinion. Every one who reported having intimate familiarity with my work (solo and with Gebru) echoed the comments above. For example, one person in the AI safety field said:

Another person who knows my work in depth said:

I find it hard to believe that Mr. Leahy knows as much about AI as he does without also knowing that you and Dr. Gebru lay out the intellectual history which he discusses in your First Monday paper and elsewhere [a point that I will return to below]. For that reason, an accusation of plagiarism seems reasonable.

Yet another person — a distinguished professor affiliated with Oxford — watched the first 25 minutes and said that, while Leahy’s analysis is similar to mine thus far, it’s not entirely clear that he’s an “intellectual thief.” I told them that the similarities become even more significant as the interview proceeds. The following day, this person sent me a second message:

You’re right — it gets worse. He says “there’s no name” for the cluster of ideologies, his cult analysis, his claim that there are but “two bits of distance” between EAs and e/accs … I don’t know what to say. Your publication record is very clear and has been picked up by mainstream journalists. I only know Leahy from the AI doc [in which he aligns with the doomer camp]. Now he must sense that the tide is turning and he wants credit for calling it out. Yes, it’s ugly. He knows your work, for sure. What to do about it is another question….so sorry…..

I will include some additional quotes just like these below.

On Twitter, I brought up the issue with Wright, who asked me to clarify what I meant by “TESCREAL thesis.”44 Here’s my abstract-length synopsis of my work (solo and with Gebru) over the past many years:

I’ve outlined this analysis in literally hundreds of articles, radio interviews, TV interviews, podcast interviews, and episodes of my podcast Dystopia Now with Kate Willett (which Leahy is welcome to appear on). Along with the Ethics of Human Extinction, TESCREALism — the history of Silicon Valley ideologies and their relationship to the AGI race — has been a primary focus of my research since about 2021, when I published my first systematic critique of EA/longtermism in Aeon. (I hadn’t coined the “TESCREAL” term yet, but I was basically talking about the same phenomenon, as I see longtermism as being, in a particular sense, the culmination of the TESCREAL movement and it’s pro-extinctionist implications.)

A few examples:

1. AI and the Threat of “Human Extinction”: What Are the Tech-Bros Worried About? It’s Not You and Me. Salon. June 11, 2023.

2. The Acronym Behind Our Wildest AI Dreams and Nightmares. (Note that I didn’t choose the title!) Truthdig. June 15, 2023.

3. What’s Behind the Race to Create Artificial General Intelligence? Understanding the TESCREAL Bundle of Ideologies. (A discussion with Timnit Gebru.) Truthdig. July 28, 2023.

4. Does AGI Really Threaten the Survival of the Species? Truthdig. June 30, 2023.

5. “Effective Accelerationism” and the Pursuit of Cosmic Utopia. Truthdig. December 14, 2023.

6. The TESCREAL bundle: Eugenics and the Promise of Utopia through Artificial General Intelligence. First Monday. June 2024.

7. What Is the “TESCREAL Bundle”? Answering Some Frequently Asked Questions. Substack. June 2024. (Note that I recently unpublished this to rewrite it. It was online for at least a year or so.)

8. The Endgame of Edgelord Eschatology. Truthdig. April 25, 2025.

9. TESCREAL. Oxford Research Encyclopedia of Science, Technology, and Society. 2026 (a prepublication draft was available by March 2026).

10. Skeptical of the TESCREAL Acronym? Read This. Realtime Techpocalypse Newsletter. July 9, 2026.

You can find more examples here, although this is not an exhaustive list. There’s also Gebru’s 2023 talk on the subject (below), as well as a 2024 conference at The New School for Social Research that centered on the TESCREAL ideologies and featured speakers like Douglas Rushkoff, whose term “The Mindset” overlaps in important ways with the TESCREAL concept.

I’ve also explained the TESCREAL thesis in multiple documentaries, including In the Belly of AI, Ghost in the Machine, and Elon Musk Unveiled: The Tesla Experiment.

As discussed in the final section of this article, I am constitutionally disinclined to complain about people “copying” or “plagiarizing” my work without attribution. The problem here is how very similar the picture Leahy paints is to the picture that Gebru and I present. There is hardly any substantive difference between his TESREAL account and our TESCREAL account.45

Why Not Cite Us?

There are three possibilities47:

1. Leahy isn’t familiar with my work on the topic. Hence, he is independently discovering my (and Gebru’s) analysis.

2. Leahy knows that my work on the topic exists but has misunderstood it. He doesn’t mention it because he considers it a flawed analysis, or doesn’t realize that his TESREAL account is, in fact, virtually identical.

3. Leahy is familiar with and understands my work on the topic, but he chose not to mention it despite outlining a virtually identical account.

Taking these in order:

1. Leahy Can’t Have Missed the TESCREAL Analysis

If (1) is the case, then I’d say that Leahy hasn’t seriously researched the topic. It’s impossible to have missed the hundreds of publications, interviews, etc. that I (and Gebru, as well) have written or given. The foundational article with Gebru already has more than 300 citations, and scholars have published articles with “TESCREAL” literally in the title (e.g., this one examining the intersection between the TESCREAL community and psychedelics).

“TESCREAL” has its own Wikipedia entry, as well as an entry in the Oxford Research Encyclopedia (linked above). People on the EA Forum and LessWrong have discussed the acronym, which some have included in the title of their articles, and Leahy has accounts on both websites (here and here). Sam Altman has referenced it on Twitter (below, LOL), Yudkowsky is aware of it, and Marc Andreessen even included “TESCREAList” in his Twitter bio in 2022. And so on.

If you write about AI, and especially if you research the ideologies of Silicon Valley, you will inevitably come across my work. Literally the first website that comes up when you search “ideologies of Silicon Valley” on Google is a 2025 paper by Kendall Giles titled, “Silicon Valley Ideologies.” Here’s an excerpt of the abstract:

Through the quest for Artificial General Intelligence (AGI), the case illustrates how intellectuals and institutions, from OpenAI to Google, sell ideologies branded as TESCREAL: Transhumanism, Extropianism, Singularitarianism, Cosmism, Rationalism, Effective Altruism, and Longtermism. These ideologies predict posthuman evolutionary destinies, mind uploading, and space colonization, with speculative existential risk frequently taking precedence over near-term social harms like bias and inequality.

Even more, I tagged Leahy in a tweet last year in which I intimated that he should cite me and Gebru when writing about the topic, as he published a 2025 report that included a section on the ideological nature of the ASI race (focusing primarily on singularitarianism).

Finally, Gebru and I are the only people who’ve discussed this specific topic in this particular way — it’s not like our analysis is a needle in the haystack. As alluded to, if you were to search for intellectual histories of the Silicon Valley worldview that’s driving the ASI race from 2022 to the present, you’d find very little except for our work. (There are exceptions, such as Elise Bohan’s PhD dissertation on the history of transhumanism, which Leahy actually cites, as well as Alexander Thomas’ The Politics and Ethics of Transhumanism.)

It is, therefore, utterly implausible that Leahy has never encountered my work during his “Internet archaeology” (his term) on the topic. As my colleague said above: “I find it hard to believe that Mr. Leahy knows as much about AI as he does without also knowing that you and Dr. Gebru lay out the intellectual history which he discusses in your First Monday paper and elsewhere.”

This was echoed by yet another person (not previously quoted), who remarked in a voice message to me:

It’s hard to say directly, “Okay, this is clearly stolen ideas.” But, I would say that the issue is that it’s kind of hard to believe, especially since he has been in the scene, or in that general ideological landscape … that he has not come across you or Timnit … that is really where the rub is. Like, he must have at some point, since he has clearly researched it, and your name is just the one that comes up first. And, in that sense, I’m just on a basic level not buying it that it isn’t plagiarized. That seems like absolute bullshit. But it’s hard to say — it’s hard to prove.

2. A Genuinely Curious Person

If (2) is the case, then I’d also argue that Leahy hasn’t done his homework. Given his TESREAL analysis, surely he would have spent some time reading my articles (solo and with Gebru). Leahy strikes me as a curious person. He seems open to new ideas and alternative perspectives. It’s hard to believe he wouldn’t have taken the time to read at least some of my papers, or check out my newsletter.

I think he’d do this even if people around him misunderstood my (and Gebru’s) analysis and insisted that my work isn’t worth looking into. It would be very odd if he uncritically took someone else’s word for it without doing some research and making up his own mind.

3. No One Wants to Think This

The final option is (3), which is obviously unethical. If (3) is the case, it would be ironic given Leahy’s (justified) screams that people in the AI race are acting unethically. I don’t want to believe that (3) is true, but the sad fact is that this sort of thing happens all the time. Indeed, I just published a long article about how Nick Bostrom has habitually taken many of his “best” ideas from other people without proper attribution. A friend of mine at an Ivy League university had a novel idea of his stolen by a peer. When I was a Special Student at Harvard, the PhD students wouldn’t share details about their dissertations with other students for fear of being plagiarized. There is no dearth of examples.

I should also note that Leahy’s organization, ControlAI, doesn’t disclose the donors behind its funding. They’re supported by “dark money,” as it were, and if I had to guess, I’d say that this money probably comes from the TESCREAL community — of which Leahy is now critical.

His previous organization, Conjecture, which he founded, was funded by TESCREALists like Sam Bankman-Fried, and even has its own dedicated page on the EA Forum, which indicates that it was deeply embedded in the TESCREAL ecosystem. So, Leahy has roots in this movement. Perhaps ControlAI is different from Conjecture in not accepting any TESCREAL money, but if it does, this would provide a strong incentive to avoid mentioning me or Gebru in any critique of Silicon Valley ideologies. That’s because most people in the TESCREAL community hate me and Gebru. (See the “Why Is TESCREAL Controversial” section of this article for discussion.)

Hence, if ControlAI is funded by TESCREALists, their funding could be jeopardized by Leahy legitimizing and promoting our work.

I See No Other Options

I cannot think of a fourth possibility. It seems that if (1) and (2) are false, then (3) must be true, and if (2) and (3) are false, (1) must be true — and so on. Of these options, (2) is obviously the best, though still not great.

If Leahy would like to clear this up, I would welcome a friendly chat — over Zoom, on my podcast, or via another platform. I have always valued collaboration, and I believe there is a lot more scholarship to be done on the topic of TESCREALism.

Why not work together in elaborating, precisifying, and improving the analysis that Gebru and I painstakingly put together and which I have extended considerably since 2023, in both peer-reviewed papers and popular media articles? Why not cross-pollinate ideas? Why not build a more robust Team Human movement?

What’s the Good News?

People Are Catching On

The good news is that, first of all, this debacle indicates that people are waking up to the TESCREAL ideologies driving the ASI race. I’ve noted this encouraging trend on numerous occasions over the past several years:

Finally!

Leahy (Apparently) Endorses the TESCREAL Thesis

The other bit of good news is that it turns out Leahy is a champion of the TESCREAL thesis. That’s cool. I’m always happy when someone joins the team (unless they’re fascists or pro-extinctionists, and Leahy is neither).

What do I want to see happen? How can this situation be rectified? At the very least, Leahy needs to mention that his TESREAL analysis is extremely similar to my analysis (solo and with Gebru). Gebru and I have been talking about the TESCREAL thesis before I can find a single instance of Leahy mentioning “Extropianism” online. Beyond this, I would welcome some sort of collaboration (doesn’t need to be formal or anything like that!). As noted above, there is a lot of work left to be done, and I suspect that aspects of the TESCREAL framework may need adjusting.

For example, Yasmeena Khan at the University of Oxford wrote a fantastic master’s thesis on the e/acc movement, titled “Coding Utopia: Effective Accelerationism & Evolving Techno-Optimism in Silicon Valley.” It offers, in part, a history of this neoteric ideology, and her account suggests that the way I understand e/acc in relation to the TESCREAL movement is not fully accurate. After reading her thesis (which is brilliant work), I came to more or less fully agree with her.

I mention this as an example of dialogue between my and Gebru’s TESCREAL thesis — which Khan generously cites — and other scholarship that builds upon and critiques it. I have no doubt that there are details that Leahy and I would disagree about, but working out such disagreements could very well push the tiny nascent field of TESCREAL Studies (or whatever one wants to call it) forward.

Unfortunately, Leahy blocked me on Twitter sometime after I sent him a DM in 2023 asking if he’d like to meet up while I was visiting London.50 I have no idea why he blocked me. It might have been in response to me suggesting last year that he’s copying my work (tweet included above). If that’s the case, it’s rather ignominious.

Fact-Checking Leahy’s Interview

I think Leahy’s analysis is missing a lot of important details. It’s like a granular, sepia version of the analysis that I present in my solo articles and with Gebru. He also said a number of things that are factually false, and — in my opinion — misunderstands the prognostications made by people like Yudkowsky in the early TESCREAL movement. Here are three examples:

1. As noted earlier, Wright says: “I think Shane Legg came up with the term AGI, didn’t he?” Leahy responds: “I think he did. Yes, if I remember correctly.” Students of this history will know that Mark Gubrud coined the term in 1997 (to the best of anyone’s knowledge). Ben Goertzel, who popularized the term, says in a paper that he got the term from Legg, though elsewhere he claims that it might have come from Peter Voss. Gubrud suggests that Goertzel and Legg very likely got it from him. Either way, Goertzel popularized it while Gubrud coined it, though Legg might have coined it independently.

2. Leahy describes I. J. Good as a “sci-fi author” who was “one of the first people to talk about the intelligence explosion.” The second assertion is true, but the first isn’t: Good wasn’t a sci-fi author. To my knowledge, he never wrote any fiction. He was instead a mathematician and cryptologist.51

3. Leahy claims that, “famously, Lord Kelvin, a very famous scientist, said that the idea of getting net energy from fission is moonshine. It’s impossible.” Lord Kelvin died 26 years before the “moonshine” comment led Leo Szilard to discover, in 1933, the idea of a nuclear chain reaction involving free neutrons. Famously, the “moonshine” statement comes from Ernest Rutherford. Leahy also says that, upon reading Rutherford’s statement, Szilard “immediately came up with a design for how to build a nuclear bomb and how it would work.” This is false. All Szilard did in 1933 was realize that if nuclei could be split by one neutron such that they release two neutrons, a self-sustaining chain reaction could result, thus liberating the enormous stores of energy bottled up in those nuclei. It wasn’t until 1938 that the nuclear fusion of uranium atoms was discovered, leading Szilard to understand right away that uranium must release neutrons in the process, thus making a nuclear chain reaction practicable. Szilard then tested this and found that uranium does in fact release neutrons. The design of the atomic bomb, based on these discoveries and insights, required the entire Manhattan Project. I wrote about Lord Kelvin (and thermodynamics), Szilard, and the making of the first atomic bomb in great detail in chapter 4 of my 2024 book Human Extinction. Fun stuff!

Finally, here’s a criticism. Leahy says:

This is what I think is so fascinating here, is that there’s a lot things [sic] are wrong, but fundamentally they [the people in these TES(C)REAL cults] did have secret knowledge which did turn out to be true. If you were in the cult sphere in like the early 2000s, you did know things about the future that other people didn’t know. So this was very intoxicating to people.

I think Leahy dramatically underestimates the causal impact these people had in bringing about the ASI race. Indeed, this is a major deficiency of his analysis, and a central feature of my analysis with Gebru. It was because of the influence of people like Yudkowsky, Bostrom, Thiel, Kurzweil, and other early TESCREALists that we ended up in our current existential snafu.

To suggest that such people were prescient is similar to congratulating me for having accurately prophesied the future when I woke up and said, “I prophesy that I will soon drink a cup of coffee!,” after which I went to my coffeemaker and made a cup of coffee — and then drank it.

Participants in the transhumanist, Extropian, and singularitarian communities (before the other ideologies were fully formed) didn’t have some “secret knowledge.” They imagined a future and then made it happen. Indeed, one could plausibly argue that no single individual has done more to inspire people to pursue ASI than Yudkowsky himself, as I’ve said on many occasions. I think Leahy’s analysis problematically misses this very important fact.

Conclusion: Why Write This?

Why do I care about Leahy not citing me (and Gebru)? Isn’t this kind of petty? Why not just ignore it?

If you’ve spent any time in academia, you’ve likely come across a type of academic — usually an older white man — who endlessly whines about others not citing them enough. They insist that this or that idea was actually theirs — that they were the first to discuss it, and hence a great injustice is committed when anyone references but fails to mention them. Such people bemoan the fact that, in their eyes, they aren’t sufficiently lionized for their supposed brilliance, insight, and originality. This often arises from an overinflated ego and/or deep-seated anxiety about their legacy.

I strongly dislike this type of person. I do not like egos, arrogance, self-importance, or narcissism. And I don’t fool myself into thinking that anyone will remember me once I’m gone.

The reason I was bothered enough to write an article about Leahy’s interview is: (a) Leahy’s analysis isn’t somewhat similar but virtually identical to mine (and Gebru’s), and (b) Leahy is a young guy with lots of money, resources, and influence; he has a larger audience than me, and boasts of direct access to political power in DC; and so on. Meanwhile, I’m an independent researcher and freelance journalist who’s currently struggling to bring in more than $12,000 a year, despite my workaholism and (what some would call) prolific output.

Neither (a) nor (b) are independently sufficient for me to have complained. But (a) and (b) together is upsetting. Those with power and influence have a duty to cite lesser-known figures whose work they draw from (especially when they’re drawing heavily). Even if one claims not to be “copying” them, one still has a duty to at least acknowledge similarities when those similarities are significant and extensive. Citing others is not hard. It takes a single breath to say, “This analysis isn’t original to me. Torres, in solo articles and with Gebru, has outlined this in detail.”

I will close with a quote from yet another friend who I didn’t quote above. Referring to this situation, they wrote to me:

Unfortunately it’s almost never that people warning early get the gratitude they deserve but instead get erased, so people who ignored their warnings can rip off their work and pretend they never ignored sometimes made fun of it.

So it goes.

As always:

Thanks for reading and I’ll see you on the other side!

3

I have repeatedly described the TESCREAL bundle as the orthodox religion of Silicon Valley (see the title of the interview from 2023 below), and characterized the TESCREAL worldview as “bizarre,” “radical,” and a “dangerous techno-utopian-apocalyptic cult” (from a 2023 article). By implication — yes, Silicon Valley is a very weird place!

7

I have also repeatedly described the TESCREAL worldview as a “religion” and “cult” (here, here, etc.). As I say in this interview from 2023:

In fact, the parallels between transhumanism and traditional religion are really quite striking. That’s really not a coincidence. If you look at the individuals who initially developed the transhumanist ideology, they were explicit that this is supposed to be a replacement for traditional religion. It’s a secular replacement.

Many other examples could be adduced.

22

I have repeatedly made this point, sometimes using nearly identical language. The TESCREAL movement is extremely elitist. They believe that they have “special access” to “fundamental truths” about what the future should look like, and that the rest of us aren’t smart enough for our input to matter. For example, in early 2022, I wrote:

In 2021, I said:

This has, indeed, been a central feature of my criticisms of the TESCREAL movement. In early 2025, I told Gil Duran:

I would say that there is an incredible amount of hubris among these people. And as someone who was in this community back in the day, I can attest that many have an idea that they have some kind of special access to fundamental truths about what the future ought to look like. And now their task is to impose this vision on the rest of humanity. So they sort of speak for all of humanity.

In another 2025 interview, I said:

The rationalist community is very influential in Silicon Valley, and what the rationalist community basically tells its members is that once you’ve mastered this or that theory within the field of decision theory, once you’ve mastered the patterns of thought that are optimally rational, you then have direct access to fundamental truths about what the future ought to look like. And because rationality provides a universal and objective perspective on the world, you don’t need anyone else’s perspective. That is exactly how these billionaires in Silicon Valley are thinking about our collective future.

In an article from around the same time, I wrote:

This was a main point of yet another EA Forum post by Cremer, bemoaning the worship of IQ. People like MacAskill, Bostrom, and Yudkowsky are elevated to an almost messianic level of cosmic importance. They are the prophets of EA-longtermism, sent by the Rationality Gods to deliver fundamental truths about the apocalyptic hazards and utopian promises that lie ahead.

There are many more examples. Leahy is parroting a central thrust of my critique: the TESCREAL movement is fundamentally undemocratic and elitist.

23

This is exactly what I’ve argued. In a 2024 interview with Tech Policy Press, I say:

It’s just worth noting that the way I understand the TESCREAL vision is as a kind of tyranny. This is a very particular view about what the future ought to look like that was crafted almost entirely by white men at elite universities like Oxford and in Silicon Valley, and now they’re trying to realize this future, and in doing that to impose this future on the entire rest of humanity. And the way they’re trying to realize this future is through AGI, because again, they believe that AGI is going to be the most important and the last invention we ever need to create. And if it’s value aligned … it’ll bring about that utopia. If it’s not value aligned, then by default it’ll kill everyone on Earth.

In a 2025 interview, I described the TESCREAL approach to building ASI as

deeply elitist and extremely undemocratic. Right now, with their billions and trillions of dollars, they are trying to create a new world run by post-humans without ever having inquired about the opinions and preferences of the rest of humanity. They’re doing this without our consent, and they don’t really care one bit about what the rest of us have to say.

In my recent “Pro-Human Manifesto,” I say:

All the AI CEOs have insisted that AGI should benefit all of humanity, but we’ll see in the second half of this manifesto that their undemocratic, tyrannical approach to developing AGI suggests otherwise.

From 2022:

I recently heard that Netflix will premiere a new series titled The Altruists on November 19 of this year. It’s about Sam Bankman-Fried and his partner in crime, Caroline Ellison, and how they committed one of the worst financial crimes in US history. The Effective Altruist (EA) movement will also, it seems, feature prominently in the series. The press release states that it’s

the story of Sam Bankman-Fried and Caroline Ellison, two hyper-smart, ambitious young idealists who tried to remake the global financial system in the blink of an eye — before they were accused of stealing $8 billion and became Gen Z’s own Bonnie & Clyde.

I hope The Altruists doesn’t present Bankman-Fried and EA in a favorable light, but describing the main characters as “hyper-smart, ambitious young idealists” suggests that it might. Furthermore, the comparison to Bonnie and Clyde raises eyebrows because the two are often romanticized as folk heroes in popular culture, and the series is being produced in part by Vox Media — which hosts a vertical called Future Perfect, once funded by an organization run by Sam Bankman-Fried and his brother Gabe! (Note that I was once a member — I am so embarrassed to admit this — of an “elite” EA group in DC. Another member, in the same room as me, was Gabe Bankman-Fried.)

Or, consider this quote from Graham Moore, the writer, creator, and showrunner of the series:

From the very beginning, I was so fascinated by both Sam's and Caroline's real stories. Those are both characters who I could not have possibly made up had I tried — I'm not nearly a good enough writer to create characters as fascinating as either Sam Bankman-Fried or Caroline Ellison. ... This form [of storytelling] allowed us to tell not just a kind of headline version of the story, but to tell what we believed was the much more fascinating, intimate, personal, deeply heartbreaking story of these two people.

“Deeply heartbreaking” — you mean for the millions of people who Bankman-Fried and his EA accomplices stole over $8 billion from? Or for poor old Sam and Caroline who just wanted to make the world better and, you know, tried a little too hard? If this series is appropriately critical of EA, I can imagine myself really liking it. But I’ll be deeply disappointed if it portrays the villainous EA movement in sympathetic terms.

I also got a chuckle out of the fact that William MacAskill will be played by a Scottish actor named Jack Greenlees. I wonder if there will be a humorous scene in which MacAskill talks to Bankman-Fried about how he considered getting the gap between his two front teeth closed “because studies showed that more ‘classically’ handsome people were more impactful fund-raisers.” Also, Barack and Michelle Obama are among the series’ executive producers, and the CEO of FTX Digital Markets is played by Matt Rife (lol). Hmm.

Rationalist Outreach Is Embarrassing

Speaking of EA, the Rationalists — a community out of which EA grew — have been experimenting with new ways of getting their AI-doomer message across. Exhibit A is a recent video from Nate Soares, coauthor of If Anyone Builds It, Everyone Dies with the doomer-in-chief Eliezer Yudkowsky. I have absolutely no idea who they think this will appeal to, as I find it unbelievably cringe-inducing. What do you think?

It’s clear they have a ton of money for “outreach,” but despite their intellectual superiority (you know, super-high IQs and near-perfect rationality), my sense is that videos like this will hurt their cause.

Another example is this music video, in which the singer, who intermittently wears a chain of paperclips around his neck, sings: “This is how we’ll die / Race to AGI / Oh oh.” The chorus consists of the repeating line: “Last year alive” — because we’ll soon “be a paperclip” and “all the value will be lost” (a reference to the estimated 10^58 digital people who could exist if we build aligned superintelligence and then colonize the universe — won’t someone think of all the digital unborn!) He even references wireheading involving “AI hyperporn.” The song closes with repetitions of: “S-T-O-P.”

As you may have noticed, Yudkowsky and Soares both make cameos, with the latter describing this song as a “banger” on social media. I suspect this — and Soares’ “big smart” video above — was filmed at Lighthaven, the Rationalist doomer compound in Berkeley operated by a group called Lightcone Infrastructure, which also maintains the website LessWrong.

You Wanna Get Food, or Something?

In my previous article, I argued that AI capabilities may be advancing, and such advancements could carry significant risks to society, but that these facts alone do not necessarily imply that AGI, ASI, or the Singularity are imminent — or even getting closer.

For a chuckle, here’s a pretty hilarious video mocking the conversational limitations of current AI systems — specifically, the unnatural and obnoxious ways that ChatGPT engages in dialogue. I have never had the urge to punch someone in the face while talking to them (even in the most annoying of circumstances), but I feel my fists clenching when I watch this:

Although very funny, it also highlights some serious questions about the extent to which AI chatbots may be influencing — subtly or otherwise — our patterns of speaking, writing, and even thinking. The worry is especially acute when it comes to children growing up with these systems.

As a friend of mine has repeatedly emphasized to me: What are the implications of AI with respect to the socialization of children? What might we expect when a nontrivial percentage of young people’s interactions involve AI rather than other humans? Even in the case of human-to-human contact, it’s likely that many others in their cohort will have been influenced by AI, meaning that even this form of interpersonal exchange may be tainted by LLM rigmarole. What should we expect from an entire generation that, during its formative developmental years, had conversations with ChatGPT like the one above?

My dad sent me another video that illustrates a similar idea — in this case, all the characters are relying on AI chatbots to tell them how to respond to others. It’s pretty hilarious:

Yes, of course, every generation has worried about how new technologies might be unraveling the fabric of society, thus dooming it to a slow deterioration until the worst dystopian nightmares have been fully realized. Taylor Lorenz recently noted in an amusing thread that novels were once “compared to opium, alcohol and cigarettes and parents wanted restrictions on children being able to read them. They wanted to raise the age to read novels to 18.”

But surely some of the concerns around AI are warranted. Many studies, for example, have converged on the conclusion that even moderate use of AI can “potentially reduce critical thinking, creativity and problem-solving.” Or, as an article in the BBC put it, “AI chatbots could be making you stupider.”

Of course, future research may disconfirm this finding, but right now the totality of available evidence clearly points toward: “We should be seriously concerned about the impact of AI on our cognitive functioning,” especially the functioning of younger generations who’ve been conversing with AI since childhood.

What do you think?

The Pigeon App

Lorenz also apprised me of a new app called Carrier Pidge. It was developed by a young man who posted on X that he “spent 3 weeks building a texting app slower than email.” When you send a message, it traverses communication systems “at pigeon speed,” meaning that it takes a message 22 hours to “fly” from Los Angeles to New York City. He adds that “sometimes the bird dies & the message isn’t delivered.” Here’s a video of the process (no audio, fyi):

You can download the app here. I did that, but can’t send a message because I currently have no friends in the “flock.” If you download it, let me know and I’ll send you a message!

The app obviously won’t catch on — it was created for a laugh — but it gestures at something most of us desperately want these days: for life to slow down a bit. Things are moving too quickly. I don’t think I’m speaking only for myself when I say that we receive too many emails and messages. The 24/7 news never lets up. Many of us are suffering from acute cases of crisis fatigue. Etc.

Although there are many aspects of modern life that I wouldn’t voluntarily give up, I also feel — as many of us do — a kind of nostalgia for the “old days” when information overload wasn’t such a pervasive problem. The Carrie Pidge app, at the very least, reminds us that there are other — perhaps better — ways of doing things.

Timnit Gebru’s New Book

Finally, I wanted to share a bit of satire from Timnit Gebru, who recently posted this on social media:

📢I’ll be creating a fully autonomous startup, staffed by AGI. I care deeply about humanity’s future and can’t wait to work alongside ethical workers to hopefully make a gazillion amount of 💰. Surely I shouldn’t be left out of the retirement and oil 💸 flowing into this bubble.

Making the money that I’m going to be making is just an unavoidable side effect of me saving the world, which I unfortunately have to endure.

Given that only I can do this (and let me just add that it’s dangerous to let anyone else attempt this), it’s unfortunate that only I will be accumulating all this wealth. But that’s the price of carrying the world’s burden and being tasked with bringing utopia to all of us.

That’s a pretty good encapsulation of how people at the AI companies think.

Note also that Dr. Gebru has a forthcoming book titled Deep Unlearning: The Rise of AI and the Radicalization of a Tech Idealist.

I’ve not read the entire tome yet, but I was given an opportunity to look over a chapter in which Timnit discusses our collaboration, which led to a 2024 paper on the TESCREAL ideologies. (For an intro to the TESCREAL concept, read this.)

That chapter was, unsurprisingly, extremely good. Timnit has an incredible ability to identify the crux of an issue, expose the core problem within it, and articulate it in powerfully insightful ways. Indeed, there have been many occasions in which I’ve messaged her (especially during our collaboration) with something like: “This statement by X seems off to me. But I can’t figure out exactly why or find the words to express what I want to say.” With envious ease, she’d shoot back a sentence or two that consistently left me thinking: “God damn, yes, that is exactly what I was fumbling over my own thoughts trying to express!”

You can pre-order the book here. I will, of course, write a review once it’s out, and Timnit will appear on my podcast Dystopia Now to discuss it. Very excited for the book release, and I hope it becomes a bestseller.

Subscribe now

PS. An Amazing Solar Eclipse

A few years ago, I witnessed my first total solar eclipse in Cleveland, Ohio. I’m not big on seeking out new experiences — oftentimes, I’d rather sit at home and read about something than go out and experience it. (Visiting new cities and countries is the one exception, although once I’m ensconced in a new place, I’m usually content to sit at a cafe reading and writing all day.)

So, I wasn’t especially excited about the eclipse, and didn’t expect much. But holy hell, it was incredible. I can’t describe with words how profound it was. The sky darkening in the shadow of the moon wasn’t anything like sunset. It was as if the sun itself had dimmed — as if I were suddenly experiencing a different sun, on a different planet elsewhere in the universe.

On Wednesday, I wasn’t in the path of totality, and it wasn’t in the middle of the day. Nonetheless, experiencing the penumbra of the moon’s shadow at sunset was still deeply unsettling in the most marvelous of ways. I took videos of the event, but won’t share them yet because they would reveal my exact location.

Instead, enjoy this video posted online, which might have been taken in Spain (I’m not sure). The cheers and shrieks from the crowd gives me chills. Incredible stuff:

Wishing everyone the very best. As always:

Thanks for reading and I’ll see you on the other side!

There is no contradiction between the following statements:

1. AI models are becoming increasingly powerful.

2. These increasingly powerful AI models pose significant risks to society, perhaps even humanity itself.

3. None of this is bringing us any closer to AGI, ASI, or the Singularity.

In other words, one can accept that AI systems are advancing quickly, these advances pose serious risks, and ASI remains as distant today as it was, say, in 2022. This article explains why.

“Much Less Weird than it Seems Like It Should Be”

As we’ve discussed on numerous occasions, many people in Silicon Valley are convinced that the Singularity is imminent. Sam Altman wrote last year that “we are past the event horizon; the takeoff has started.” In January, Musk posted on X that “we have entered the Singularity.” He reiterated this in early August, after news about OpenAI’s Astra model (discussed below) broke, saying:

(Since he asked, the temperature over here in Europe is UNBEARABLE. Good thing that AI isn’t contributing to climate change!)

Demis Hassabis, who recently stepped down as CEO of DeepMind, said that “ten years from now, I think we’ll realize that we were standing in the foothills of the singularity now.” He believes ASI will arrive by the early 2030s, after which we will quickly start building a Dyson swarm around the sun (lol). And Dario Amodei claims that AGI could arrive later this year. Even Jensen Huang of NVIDIA said last March: “I think we’ve achieved AGI.”

Amusingly, after Altman proclaimed that we’re “past the event horizon” in his blog post, he added that “humanity is close to building digital superintelligence, and at least so far it’s much less weird than it seems like it should be.”

Yeah, dude, it’s not that weird because the Singularity isn’t happening! The event horizon still looms above the actual horizon before us — which is to say, we’re nowhere close to triggering an intelligence explosion.

The OpenAI-Hugging Face Incident

Yet, it’s clear that AI models are becoming significantly more “powerful,” and that newer models — Mythos being the most obvious example — are introducing novel risks that should worry us greatly.

I’m sure you’ve heard of this incident — it’s one of several that have recently been reported by AI companies. Basically, OpenAI was testing “an experimental in-house AI, working with the latest commercially available model.” As part of the test, they gave it “a set of cybersecurity tasks” to complete. The AI then figured out a way to escape its digital sandbox by identifying and exploiting “a previously unknown zero-day vulnerability,” hack into Hugging Face, and search Hugging Face’s datasets for a solution to the cybersecurity problems it was tasked with solving.

This happened in part because OpenAI failed to design an adequate sandbox. Some of the AI’s security guardrails were turned off — if they had been on, they “may have prevented this” from happening — and the sandbox “had one small connection to the outside world, to allow the AI to download software it needed.”

This portal to the public Internet was mediated by a “package registry cache proxy,” which is basically an intermediary server that provides a kind of barrier between the AI and public Internet. But the AI found a vulnerability in this intermediary server and, in doing so, broke free of its prison, thus enabling it to access the Internet and hack into Hugging Face.

It’s worth highlighting that no one at OpenAI told the AI to cleverly break out of the sandbox and hack into another company — which constitutes a felony! Yet, as Gary Marcus notes:

the current incident was NOT an attempt where [the] system built a goal for itself or developed a motive; the system was following instructions, but not setting high level goals. It was not trying to take control of the world a la Terminator, it was just trying to cheat on a test, which is at least a bit less scary.

“On the less comforting side,” he adds that OpenAI’s guardrails (when turned on) are nonetheless

likely to be permeable, just like all guardrails anybody has built to date. Even putting aside the thorny questions of open weight systems, we have no guarantee whatsoever that future models won’t be able to do similar things, such as finding zero-day exploits to hack systems. To the contrary, we can expect more incidents of this type.

This is unsettling. As a New Yorker article puts it, what might have happened “if the rogue AI had had access to a robot, or a self-driving car, or a bio lab, or a weapons system”? Recall that Pete Hegseth is actively transforming the US military into “an ‘AI-first’ war fighting force across all components, from front to back,” according to an official document from the Department of War. Already, war game tests show that AIs will threaten nuclear war in 95% of scenarios.

Even more, Chinese companies are releasing open-source AI models that, as such, allow people to remove any guardrails they might come with. For all of these reasons, “we can expect more incidents of this type,” though we should also expect future cyberattacks to become more sophisticated and difficult to prevent.

Subscribe now

AI Invents New Viruses

Around the same time this snafu became public, the New York Times reported that

Scientists at Stanford University and the Arc Institute, a research organization in Palo Alto, Calif., taught AI [called “Evo,” similar to ChatGPT] to recognize patterns of DNA structure in nature, and then to use that data to write recipes for entirely new viruses.

The researchers followed those recipes to create DNA molecules, which they inserted into bacteria. The modified bacteria then produced viruses never seen in nature. The viruses were able to infect other bacteria, demonstrating that they were viable. …

As Mr. King and his colleagues tested more genomes, they saw more clear dots. All told, they discovered that 16 of Evo’s genomes produced viable new viruses.

They proved to be as resilient as natural ones. In fact, some multiplied faster than Phi X-174 [see below]. “They’re not just sickly versions of stuff that already exists,” said Oliver Crook, a protein chemist at the University of Oxford who was not involved in the new study.

The article adds that, fortunately, “the viruses dreamed up by A.I. do not pose a threat to humans, because they are all similar to a naturally occurring virus called Phi X-174, which can infect only bacteria,” and they’re “not radically new creations. They tend to be very similar to natural species, relying on the same underlying biology.”

Nonetheless,

Evo’s initial success has raised concerns that A.I. could be used to create deadly pathogens. “You could say, ‘Hey, genomic language model, make me an influenza genome that is modified to be more transmissible or to be more lethal,” Dr. Hanke speculated.

The potential dangers were already on the minds of the researchers as they trained Evo. They did not provide the model with data about viruses that infect humans, and they excluded similar viruses that infect other animals, plants and fungi.

The possibility of AI-enabled designer pathogens has been discussed for decades among scholars of global catastrophic risks. An express goal of synthetic biology is to “black box” aspects of genetic manipulation: rather than spending years in a laboratory acquiring “tacit knowledge” about how to manipulate the building blocks of life, it aims to enable individuals to achieve the same ends by simply pushing a button. (This is also called “deskilling.”)

Furthermore, the genomes of horrific pathogens like Ebola and smallpox are publicly available online (I won’t include a link!), and there’s no theoretical barrier to combining the incurability of Ebola with the contagiousness of the common cold, lethality of Rabies, and long incubation period of HIV. Doing so would enable “risky agents” (a technical term meaning those with an explicit desire to destroy civilization and/or humanity) to synthesize a germ that spreads around the world undetected, infects nearly everyone, and then 10 years later results in mass death, perhaps even human extinction.

Is this hyperbolic? Maybe, a little — but the point is that AI could supercharge the risk. If risk = consequences x probability, then even a low-probability pandemic that kills a large fraction of the human population would still constitute a huge risk.

Add to this the fact that there is a very large number of people — a small percentage of the total population, but a sizable demographic in absolute numbers — who harbor a death wish for humanity. I’ve written about some of these people in the past, such as the Efilists, who say they’d murder everyone on Earth, including themselves, to eliminate all suffering. (This arises from their allegiance to negative utilitarianism.) An Efilist even bombed a fertility clinic in California last year.

The point is that initiating an engineered pandemic is becoming increasingly feasible. Deskilling, publicly available genetic data, tools like CRISPR-cas9, and now AI are a recipe for disaster.

Subscribe now

AI and Mirror Life

What worries me even more than designer pathogens is the possibility of AI assisting in the creation of “mirror life.” This refers to forms of life that consist of molecules with the opposite chirality of the molecules that comprise all biological life on Earth right now. A bacteria with mirror molecules could potentially infect the human body without being detected by our immune systems — such bacteria would, essentially, be invisible to our internal defenses.

As an article in Science, written by nearly 40 prominent scientists, declared:

We cannot rule out a scenario in which a mirror bacterium acts as an invasive species across many ecosystems, causing pervasive lethal infections in a substantial fraction of plant and animal species, including humans.

How might AI exacerbate this risk? Amodei himself highlights the danger in his essay, “The Adolescence of Technology,” writing that a “sufficiently powerful AI model (to be clear, far more capable than any we have today) might be able to discover how to create it much more rapidly — and actually help someone do so,” where “it” refers to “mirror life.” Yikes.

AI Capabilities are Getting Spikier, Not More General

All of this leads to the linchpin of my argument here: advances in AI and the potentially catastrophic risks they’re introducing don’t for a moment imply that we’re getting closer to AGI, ASI, or the Singularity.

This is counterintuitive, but consider a recent X post from a professor at Cambridge named Adam Hunt. He reports having previously been bullish about AI, but now considers himself bearish. The reason is that

the whole “it turns out if you keep training and scaling the models more they develop broad new capabilities in lots of domains” thesis is wrong (sorry Demis). The recent batch of models haven’t got more general, they’ve got less general. This is most obvious in the fact that their language outputs have got much worse in comparison to e.g. o3. If they were gaining generalist capacities we would expect them to be describing their work in ever more graceful and comprehensive prose!

Take a look at this widely shared image:

The “mainstream AGI thesis” on the top hasn’t panned out so far. On this account, AI will initially show superhuman capabilities in narrow domains, but as research advances, its capabilities in other areas will also increase. Eventually, the AI will exceed humans in all cognitive domains of interest, at which point we’ll have ASI.

In fact, what appears to be happening is that AI is getting (significantly) better in narrow domains (e.g., coding, finding software vulnerabilities, inventing new viruses, etc.) without getting much better in all other domains — i.e., those it currently underperforms in. The spikiness of its capabilities is getting spikier. If the trend continues, it will become ever-more superhuman in particular domains without attaining general intellectual capabilities.

When someone asked about whether “chain-of-thought make[s] AI more general, less general, or the same?,” Hunt responded:

I dont know if ‘less general’ or ‘more general’ is the dimension that chain of thought works upon.

I think chain of thought basically exaggerates existing potency in the underlying models, by allowing that existing potential to reflect and re-reflect with its existing capabilities. So with coding that leads to much much better coding, because the models are good at coding.

But in scientific logic, which LLMs are not good at (e.g. in my work, recently Opus 5 tripped up over how seriously should it take one specific study of a particular drug’s effects on ADHD) then chain of thought doesn’t help - it gets trapped in illogical loops that it self justifies and can’t leave alone.

So no, I don’t think that scaling up inference compute will suddenly get you to AGI, because of fundamental spikes and flaws in the underlying training data/RL which lead to blind spots or weaknesses in the basic model, which can just compound over time in chain of thought rather than getting suddenly fixed.

This seems right to me. One can thus acknowledge major advances in AI capabilities while simultaneously holding that these advances aren’t inching us closer to ASI or the Singularity.

Subscribe now

The Fallacy of Composition

Marcus makes a related point in a recent blog post, highlighting how AI hypesters are committing the fallacy of composition. For example, OpenAI reported that its new (unreleased) AI model called Astra “solved 10 major open problems in mathematics, quantum complexity, and theoretical computer science.” That led Dean Ball — among many others — to declare that:

The fallacy of composition occurs when “one infers that something is true of the whole from the fact that it is true of some part of the whole.” In the case of AI, the fallacious reasoning goes: “AIs are becoming more generally intelligent — thus getting us closer to ASI — because they are becoming more capable in this or that particular, narrow domain of cognition.”

That absolutely doesn’t follow. It’s like saying, “A particular aspect of my life is improving, as I now have more free time during the day to pursue hobbies. Therefore, my entire life is improving” — which of course might be false if one also just lost their job, spouse, etc. The increasing spikiness of AI capabilities doesn’t imply that more recent AI models are closer to ASI — superhuman general intelligence — than previous ones.

Conclusion

What’s the takeaway here? It’s that the incidents mentioned above don’t make me — and shouldn’t make you — more concerned that Altman, Musk, and the others are right in proclaiming that the Singularity is imminent or has already started.

At the same time, they should make us more anxious about the potentially catastrophic consequences of increasingly powerful spiky AI systems. An AI-enabled pandemic could be devastating. AI-enabled mirror life could literally destroy the biosphere. Cyberattacks will become more difficult to anticipate and prevent. And the fact that the US military is actively integrating AI into its infrastructure is hugely worrisome (as you may recall, Anthropic’s Claude was used in the illegal kidnapping of Nicolás Maduro and has helped select targets in Iran). Etc. etc.

So, I find myself both deeply worried about these new developments and no more nervous than I was about ASI suddenly emerging from the bowels of an AI laboratory. As I like to say:

Subscribe now

But what do you think? What have I missed? How might I be wrong? As always:

Thanks for reading and I’ll see you on the other side!

As you may know, some of the authors behind the widely-discussed “AI 2027” paper published a follow-up titled “AI 2040.” One of the authors, Daniel Kokotajlo, was a buddy of mine when I lived in North Carolina. He even read a draft of my 2017 book on existential risks and provided comments:

Whereas AI 2027 focused on how things could go terribly wrong with ASI (artificial superintelligence), this new document outlines a “positive vision for what should happen instead.” Both documents contain little more than wild speculations paired with unfounded assumptions and unrealistic predictions. In the Postscript, the authors write:

We think the world is asleep at the wheel. People say the words “AI will be transformative” without thinking concretely and seriously about the implications of broadly superhuman AI.

Yet I see no evidence that the authors have thought seriously about the implications of their best-case scenario, which they call “Plan A.” Plan A is little more than the TESCREAL vision of the future and, if realized, it would likely have devastating consequences for humanity — for reasons I outline in my “Pro-Human Manifesto,” and which I’ll briefly touch upon below.

To give you a sense of the normative futurology behind Plan A, here’s how they describe “Life after ASI.”

The resulting plans are too many to number. There are all sorts of schools of thought about what to do with your galaxy of resources,1 including:

• Defer to the Future: Some people decide to give most of their resources to their children (or other people in the future), and let them choose.

• Human Flourishing: Set up worlds of normal, free people living ordinary lives. This is a harder problem than it first seemed, because human civilization has already become profoundly abnormal: by this point, human activities like art or science has already become dominated by machines. Some handle this by accepting that their galaxy will be filled with post-scarcity humans, able to pursue whatever interests they find most meaningful. Others decide to roll back technology to leave space for human labor.

• Digital Human Flourishing: If humans living happy, flourishing lives is good, then why not increase the number of people living these lives? Brain emulations can be run for vastly cheaper than real humans: a planet-size computer could plausibly simulate the equivalent of a million planets with happy civilizations.

• Acausal Trade: Some argue that there likely are aliens in distant unreachable galaxies and that we could use various mechanisms to make deals with these aliens. If this is true, they argue, we could cut a deal to pursue compromise values rather than narrowly maximising our own values. Therefore, we might, on net, be able to do the most good by pursuing not just our own values, but instead a compromise of a vast number of other civilizations.

Humanity spreads across space in a dizzying variety of forms and ways of living — more diverse than anything Earth alone could have produced.

The statements above are consistent with what Kokotajlo has said in interviews:

Here are a few reasons I strongly dislike the “AI 2040” analysis:

1. Extreme Hubris

They write: “First, our goal was to construct a positive vision, a grand plan for how to achieve an actually good future for everyone.” The extreme hubris of this statement is difficult to overstate. If I may use some vulgarities to get the point across: Who the fuck are they to be declaring that they know what “an actually good future for everyone” looks like?

(Sam Altman himself once said something similar when lying about how OpenAI will include people from around the world on its board. He said: “We’re planning a way to allow wide swaths of the world to elect representatives to a new governance board. Because if I weren’t in on this I’d be, like, Why do these fuckers get to decide what happens to me?”)

This is classic TESCREAL: the entire movement is infatuated with the supposed “intelligence,” “rationality,” and “brilliance” of its members. This enables them to believe that, because of their “intelligence,” “rationality,” and “brilliance,” they can speak for everyone in outlining a grand teleological vision of our collective cosmic future. These people think they have special access to fundamental truths about what the future ought to be, and by virtue of this prophetic knowledge their vision is superior to that of the benighted masses. It is a kind of messianic hubris, a radical form of elitism — nothing less.

In contrast, consider my favored approach to, as it were, designing a genuinely good future for everyone. I would argue that no single group should have the power to unilaterally impose their vision on everyone else. Rather, our collective future should, from the very start, be determined through a thoroughly democratic process. The people — you and I — should have a say in how things turn out. We should have a say in how advanced AI is developed, and whether it’s even developed in the first place.

As my friend Monika Bielskyte likes to say, “You cannot design for, you must always design with.” That succinctly captures the essence of my position, and there is not a single trace of this democratic approach in the “AI 2040” document.

Nowhere do they acknowledge that people outside the dumb TESCREAL movement should have a say in whether superintelligence is birthed in the bowels of an AI laboratory. Nowhere is there any indication that the authors are even curious about alternative visions of the future. The authors, I think, are completely unaware of just how much their preferred vision has been intimately shaped by the values of the milieu of Western capitalism plus utilitarian ethics (as I’ve written before, these are essentially the same). Maybe they don’t care.

To be clear, I am not saying that the authors shouldn’t express their opinion about which future is best. The whole point of establishing a governance mechanism to democratically determine the future is for people — or representatives — of every nation, tribe, religion, tradition, race, sex, gender, socioeconomic demographic, political party, ideology, and so on, to have a voice in the process of future-designing. That includes the voice of Kokotajlo and the other TESCREAL utopians.

But this is not the point of Kokotajlo’s vision. He and his coauthors are laying out a roadmap for the AI companies, policymakers, etc. to follow: slow down the ASI race, solve the “alignment problem,” and then create their version of a cosmic “utopia.”

Imagine, for a moment, that a group of religious extremists in Iran announce they’re building a new technology that could determine the entire future of humanity. There’s a good and bad way to build this technology, they claim, and Iranian scientists are working hard to ensure that it’s built the “right way.” These people then outline a roadmap for the future based on Islamic principles, which they declare will be “good for everyone.” The obvious response is to shout: “Who the hell are you to speak for everyone?” Yet, there is no structural difference between what the “AI 2040” authors are doing in their document and what these religious extremists are doing in this fictional example.

I genuinely can’t comprehend the level of hubris underlying “AI 2040.” Indeed, the hubristic heart of the TESCREAL movement is one reason I left circa 2019. These people have no idea how big the world is, yet feel wildly confident in their ability to design beneficial-for-all futures. Nonsense.

2. Climate Change

In their preferred future, Plan A, there is literally no consideration of climate change. Not once do they mention global biodiversity loss or the sixth major mass extinction of the last 3.8 billion years. How can you present yourself as a serious scholar of the future and not mention the environmental crisis?

I’m in the foothills of the Alps right now, and yesterday it was 97F (36C) — roughly 20 degrees over the average high for August. Half of the world is on fire. And 2027 is predicted to shatter temperature records around the world (many of which have already been broken this year) due to a historically unprecedented Super El Niño. By 2040, large parts of the world will be more or less uninhabitable — yet the authors say literally nothing about this.

Why? The main reason concerns “magical thinking” about ASI paired with an “engineering mindset.” These people simplistically see all problems as engineering problems — that’s why Nick Bostrom talks about a “solved world” after ASI arrives. If we have a superintelligence, then we automatically have a super-engineer; and if we have a super-engineer, it will be able to solve every engineering problem facing us — which is to say, once again, all problems. (Bostrom even talks about “paradise-engineering.”)

Climate change is no different. The only reason we haven’t solved climate change, on their account, is a lack of intelligence (and hence engineering). Once we have superintelligence, it will find some brilliant technical solution and immediately reverse all ecological pollution, restore ecosystems, remove all excess CO2 from the atmosphere, and so on.

This is magical thinking. The truth is that a lack of intelligence is not the reason climate change hasn’t been solved. Building a superintelligent machine isn’t going to somehow fix the problem before 2040. By then, Earth will be burning, millions or billions of people will be dying or migrating, and entire societies will be collapsing. It’s truly abysmal that the authors don’t take climate science seriously.

3. No One But the TESCREALists Want AI to Run the World

The authors write that in 2033,

as AI capabilities improve throughout Plan A, they increasingly shape every facet of life. … AI agents now form a population of two hundred million virtual workers that think and act 50x faster than humans, and never sleep. … By early 2036, there are 200 million AIs, equivalent to a workforce of around 100 billion humans, and 2 billion robots with some mixture of humanoid and other specialized form factors. … Gradually more institutions and equipment are handed over to AIs, such that it’s no longer true that humans could shut it all down if they wanted to. In fact it’s extremely untrue: Soon many of the world’s militaries are autonomous, run by AIs sworn to uphold various constitutions and treaties.

Why on Earth do the authors assume that people won’t fight back against the growing dominance of AI and attendant disempowerment of humanity? Of course there will be mass protests, civil unrest, and outrage. Depending on the details, there may be violent conflict in the streets as Butlerian jihadists launch a war against the ascendant AI overlords taking over the world.

This is a major weakness in the authors’ analysis, and you can count me in as one of the pro-human fighters who will do whatever’s necessary, within the bounds of my partly deontological ethics, to prevent the situation described above. Frankly, the authors’ vision of things going right sounds like a dystopian nightmare to me.

4. Colonizing Space Would Result in Constant, Catastrophic Wars

The authors blithely assume that space colonization won’t be a catastrophe. They completely ignore the academic literature on this topic, with renowned political scientists like Daniel Deudney arguing (in Dark Skies) that the almost certain outcome of colonizing our solar system will be constant, devastating conflicts. I’ve written about this topic numerous times, such as here, though I would recommend the closing chapter of Deudney’s excellent book for the original, robust analysis.

Again, this is an example of magical thinking — or what I call the utopian mindset. This mindset gives utopians a license to ignore the messy details of the world by saying, “I don’t know how exactly this will happen. But future technology — especially machine superintelligence — will have magical powers that will somehow enable us to figure it out. Hence, we needn’t worry our pretty little minds about it right now.”

This phenomenon is similar to the “suspension of disbelief” in fiction, whereby one ignores the plausibility of details in a story. No one watches Game of Thrones and stops to ask, “Wait a moment! Are the wings of those dragons really large enough to carry their massive bodies through the sky?” That’s an absurd question.

Similarly, those who embody the utopian mindset habitually dismiss questions about how exactly ASI will reverse climate change, upload our minds to computers, and colonize the universe without triggering catastrophic wars as absurd.

5. ASI Is Fundamentally Uncontrollable

The authors assume that there is a way to control ASI. We just haven’t discovered it yet. That’s why we should slow down ASI capabilities research: to give us time to solve the “alignment problem.”

But why think that ASI could ever be controllable? We’re talking about a perpetually evolving computer program with, by definition, far greater cognitive abilities than all of humanity combined. And yet we will somehow still control this infinitely self-improving beast?

Roman Yampolskiy (another guy I knew from the TESCREAL community!) is probably right that this is impossible, in the very same sense that a perpetual motion machine is impossible. My friend Remmelt Ellen has written about it here.

6. More Magical Thinking

The authors write that around 2036,

many of the world’s evils have dramatically reduced. Malnutrition, lack of medicine, and homelessness are nearly banished. Many diseases have been cured. Crime rates are lower than ever before. … People are wealthier and have more free time, plus there are new technologies that help, like AI matchmakers, better medicine, and AI tutors. And of course even the poorest people can now afford exotic vacations, amazing games, and enthralling entertainment.

Um, okay? Again, magical thinking enabled by the engineering and utopian mindsets. No serious discussion of how the tech elite will exploit ASI — if controllable — to permanently entrench their positions of power, control, and domination in society. The authors need to do a lot more to convince thoughtful readers that this is even remotely plausible.

7. There Is No “View from Nowhere”

The authors write about 2036, just 10 years from now!:

There are still problems in the world, and you can still contribute to solving them, partly by donating or volunteering but primarily by being politically active. Your vote is your most important asset.

The honest AI forecasters are helpful here. Years ago, if an AI said that one presidential candidate was better than another, people would suspect bias and the embarrassed company would retrain the AI to evade such questions. Now, thanks to transparency and improved alignment techniques, there are much smarter AIs that people can see don’t have any biases trained into them, that have built up an excellent track record over several years. When they weigh in on policy questions, people listen, especially when different AIs trained by different companies converge to the same answer.

AIs with no biases! Think for a moment about what that means. It assumes that there is a perfectly objective view from nowhere that, as such, is completely free of biases. The problem is that there is no view from nowhere. All views are views from somewhere. Not even science is perfectly objective.

Imagine asking the AI whether one should vote for a presidential candidate who’s a fascist and white supremacist. What would an unbiased answer to this look like? There is no possible response that isn’t biased in some way.

For example, a “no” response would be biased toward views that I hold, e.g., that fascism and racism are bad. I consider those biases epistemologically and ethically justified, which is why I hold them. But there’s no mind-independent fact of the matter that one can point to in showing that my biases against fascism and racism are correct. There are only more or less compelling arguments.

Later on, the authors write that

during the 2036 election season, voters are unprecedentedly well-informed, and the politicians that win have a genuine commitment to responsible stewardship of the upcoming singularity.

This assumes that all voters want to be well informed. Why think this?

Even more, studies show that climate deniers in the US are often just as knowledgeable about climate science as those who accept the climatological consensus (i.e. lack of education isn’t the best predictor of denialism — ideology is), meaning that being more informed doesn’t imply that voters will make wise or judicious decisions at the voting booth.

8. Woke AI?

The authors write:

Unless the alignment science is somehow all wrong, typical AIs are now more virtuous than the most virtuous humans. This alone has profound effects: it’s as if saints and angels were walking among us.

What on Earth do they mean by “virtuous”? Are they saying that future AIs will be “woke” — because that’s how I’d understand “virtuous.” This is just inane drivel.

9. Butlerian Jihadists, Unite!

They say that in 2039:

AIs have become increasingly load-bearing in all facets of society, from business to politics to even (some parts of) the armed forces.

Again, the vast majority of humanity will likely revolt against this. Most people aren’t going to want AI to take over “all facets of society.” More probable is that there will be blood (and microchips) in the streets as the Butlerian jihadists fight against the AI takeover.

Conclusion

I could go on, but it would be tedious. “AI 2040” is not a work of serious scholarship. It’s not written by genuinely thoughtful people who are aware of their own biases — of the extent to which their “utopian” vision for the future is rooted in white, Western, capitalist values. (Indeed, all the authors are white men. Any surprise there?)

The document is severely lacking in viewpoint diversity, and the hubris behind their prescriptions for the future is staggering. It’s a symptom of how impoverished the mainstream debates about AI and humanity’s future that documents like this get attention.

What have I missed? What do you disagree with here? How might I be wrong? Please share your thoughts below! As always:

Thanks for reading and I’ll see you on the other side!

He suggested that I start a weekly newsletter. And so, on August 3, 2025, I published my first Realtime Techpocalypse article. Since then, I have published 107 articles, which amounts to slightly more than 2 per week. I am deeply indebted to Gil’s advice — if not for him, this newsletter wouldn’t exist! The least I can do in return is to implore those reading this to buy a pre-sale copy of Gil’s forthcoming book, The Nerd Reich. I haven’t read it yet, but I have absolutely no doubt that it’s amazing. Indeed, given the press that Gil has so far received, it looks like it will be a smash hit. If you have the money, please consider purchasing it asap! Pre-sales really help.

Thank you so much to everyone who’s subscribed to this newsletter. I am incredibly grateful for your interest in my thoughts about technology and, well, all-things-relating to the end of the world. Extra special thanks to those with the means and willingness to pay for this newsletter. It is not profitable opposing Silicon Valley and the TESCREAL ideologies. There are lots of anti-AI TESCREAL folks who are rolling in money from Open Philanthropy, Jaan Tallinn, and other funding entities. But there is almost zero funding for anti-AI folks who are also anti-TESCREAL.

Yet, in one year, I went from earning nothing from this newsletter to bringing in nearly half of the $20k that I need to pay all of my bills. I am deeply appreciative of your financial support, and if you have an extra $7 to spare each month, please consider becoming a paid subscriber. You can also support me on Patreon here, or on GoFundMe here. As I detail in “Why I Fled the US,” I spend virtually all of my money on three things: food (and I never eat out), accommodations (~$700/month or less), and a train ticket once a month to nomadically wander to a new city in Europe (~$60). I am basically an ascetic monk, lol.

Here’s a curated list of highlights from the last year, after which I’ve included a few poll questions about how you’d like to see this newsletter grow and evolve. Thanks again for making this possible, friends.

Subscribe now

The TESCREAL Ideologies (Intellectual History of Silicon Valley):

1. Skeptical of the TESCREAL Acronym? Read This. Probably the most accessible introduction to the TESCREAL thesis that I’ve written (even more so than my original intro published in 2023). “TESCREAL” is a term of intellectual history. It names a constellation of overlapping and interrelated ideologies that have given rise to the Silicon Valley worldview. This worldview, in turn, has launched, sustained, and accelerated the ongoing race to build God-like superintelligence. Highly recommended.

2. Is Transhumanism a Religion? Yes. This article explores how transhumanism was originally introduced as a secular replacement for traditional Christianity. It discusses how the Singularity is analogous to the rapture, and how transhumanism has its own version of resurrection: cryonics. I discuss many more similarities. Transhumanism — the backbone of the TESCREAL bundle — has become the orthodox religion of Silicon Valley.

3. Three Lies Longtermists Like To Tell About Their Bizarre Beliefs. These three lies are: (1) Longtermists care about avoiding human extinction. In fact, they only care about avoiding final extinction (the failure to create posthumanity), not terminal extinction (the disappearance of our species). (2) Longtermist care about the long-term future. If maximizing value requires a shorter rather than longer future, they will choose the shorter future. And (3) longtermists think that future people matter. In fact, people do not matter for their own sake, on the utilitarian-longtermist view. They are mere means to an end — that’s it.

4. Effective Altruism Is a Dangerous Cult — Here’s Why. Even people in the EA community worry that it’s a cult. I believe it’s a cult because I spent many years as an EA, and saw first-hand how cultish it is. Some call it the “Scientology of Silicon Valley.”

5. How Effective Altruists Use Threats and Harassment to Silence Their Critics. This re-examines some of the threats of physical harm and online harassment I’ve received from the EA cult for daring to challenge their views.

6. TESCREALism and the Race to Build an AI God. This introduces my Oxford Research Encyclopedia article on “TESCREAL.” I also argue that “if TESCREALists were to successfully create [their version of] utopia, this itself would be utterly catastrophic for humanity.” The most likely outcome of “utopia” would be (terminal) human extinction.

7. Is AI Apocalypticism Really that Different from Believing in the Rapture? Examines the similarities between religious and AI apocalypticism, i.e., the belief that the world as we know it will soon end. Spoiler: they are very similar!

8. From Trump to TESCREAL, Eugenics Is Everywhere. Eugenics is an idea that just won’t die. This article explores how it’s alive and well within the MAGA movement, as well as Silicon Valley (via the TESCREAL worldview).

   Subscribe now

Pro-Extinctionism in Silicon Valley:

1. One Half a Pro-Human Manifesto. This is essentially chapter 4 of my forthcoming book, Will Humanity Survive? I consider it one of the most important essays I’ve written recently (insofar as anything I write is important). It adduces a plethora of shocking examples in arguing that pro-extinctionist sentiments are ubiquitous within Silicon Valley, even among those who report being on the side of “humanity.” It also argues (in more detail) that the most likely outcome of realizing the utopian world envisaged by TESCREALists is that our species will die out. This, along with #1 above, is essential reading in my opinion.

2. The Perplexing Rise of Antinatalism in Silicon Valley. This examines the strange coexistence of pronatalism and antinatalism within tech culture. It introduces the term “replacement antinatalism,” in contrast to “traditional antinatalism,” to describe the growing demographic of young people who believe it’s “fundamentally unethical” to have children because AI will soon rule the world. Keep your eye on this trend moving forward!

3. If Anyone Becomes Posthuman, Everyone Dies Out. Explores what I call the “coexistence” and “replacement” views, which concern our potential future relationship with posthumanity. I show that the replacement view (i.e., pro-extinctionism) is widely held within Silicon Valley, and argue that the coexistence view will almost unavoidably lead, in practice, to the extinction of our species. Hence, if we want to preserve Homo sapiens — and I claim that we should — we must oppose the creation of posthumanity in all its forms. If anyone becomes posthuman, every human being will perish soon after.

4. The Scariest People in the World: Maniacs, Misanthropes, and Silicon Valley Pro-Extinctionists. Discusses “agential risks,” i.e., the risks arising from agents who would willingly destroy the world if only they could. Building on my earlier academic work (here and here), I add a new category to the list of risky agents: TESCREAL pro-extinctionists who want to replace humanity with AI.

5. Sam Altman: Human Extinction Is Our “Best-Case Scenario.” Discusses my Truthdig article in which I point out that Altman argues our “best chance” of surviving the advent of superintelligence is to “merge” with AI. In other words, the only way to avoid (one type of) extinction is to undergo (another type of) extinction. (See, again, the difference between “terminal” and “final” extinction.)

6. Will Superintelligence “Usher in a Heavenly Golden Age”? Silicon Valley Thinks So. Examines, among other things, the fact that “many people at Anthropic believe that they might be making the next species to succeed us. That maybe humans don’t live after that, and so it’s really important to give Claude good values” — to quote Holly Elmore. Wild stuff!

7. Intro to Pro-Extinctionism (parts 1, 2, and 3). Part 1 examines the “worthy successor” ideology of Daniel Faggella. I show that this ideology is deeply nihilistic. (See this follow-up for reasons his ideology is incoherent and self-defeating.) Part 2 and 3 are less notable, as they’re essentially early drafts of “One Half a Pro-Human Manifesto.”

   Subscribe now

On Specific TESCREALists:

1. Nick Bostrom Took Many of His “Best” Ideas from Other People. It turns out that the TESCREAL luminary, Nick Bostrom, has habitually borrowed ideas from others without proper attribution. I provide 8 glaring examples, though there are many more than this. As the silly saying goes, “good artists copy; great artists steal.”

2. Eliezer Yudkowsky’s Long History of Bad Ideas. Yudkowsky is lauded by people in the cult he founded — Rationalism — as being a genius. He’s even repeatedly described himself as such. Yet he holds some incredibly stupid views: that almost all nonhuman animals have no conscious experience, and hence cannot feel pain; that it probably wouldn’t be a “bad thing” for a child under the age of 6 to die; that he’d be willing to sacrifice “all of humanity” to create “god-like superintelligences … having fun”; and so on. There are good reasons you should have zero respect for the guy.

3. The Political Power of Eschatological Thinking: This Is Why Peter Thiel Is Talking About the Antichrist. Argues that “a true dedication to and belief in eschatological narratives can mobilize people like no other ideology or worldview can, while ‘justifying’ virtually any act to protect, preserve, and accelerate the arrival of paradise — including extreme measures, violence, and even genocide.” Hence, “by tapping into a rich reservoir of powerful images embedded within our collective imagination, images that have played a crucial role in some of the most consequential events of human history, Thiel may hope to mobilize people to join his pro-libertarian fight against the unholy forces of ‘big government.’”

4. Is Sam Altman a Sociopath? — “Highly toxic.” “Psychologically abusive.” “A person of low integrity.” This is how Altman’s colleagues have described him. I also discuss the accusations of abuse made against Altman by his sister, Annie.

5. Preparing for the Singularity: An Ethnographic Survey of AGI Apocalypticists. A fascinating look at how people are preparing for the Singularity. Some have lost all motivation. Some are spending their retirement savings now because they don’t think money will have value in the near future. Others are working extra hard to improve their health based on the belief that if they live long enough to witness the Singularity, they will live forever. A good resource, I think, for historians 50 years from now.

6. The Moral Cowardice of Leading Effective Altruists (and How EA Has Probably Made the World Worse). Argues this: “Claim: EAs are exceptionally ethical people. Prediction: Leading EAs will, therefore, be the loudest voices on the vanguard of speaking out against atrocities like the Gaza genocide, ICE’s brutalization of immigrants, the rise of American fascism, etc. Observation: Crickets from the EA leadership.” Sigh.

7. Eliezer Yudkowsky and the Unabomber Have a Lot in Common. Discusses an interesting chapter of intellectual history: Kaczynski’s manifesto, Bill Joy’s 2000 “Why the Future Doesn’t Need Us,” the transhumanist opposition to Joy’s position (exemplified by Ray Kurzweil), and the way Yudkowsky’s views have shifted over time from aligning with Kurzweil’s to aligning with Joy’s — the latter of whom was partly inspired by the manifesto of Kaczynski.

8. Nick Bostrom’s Pro-Superintelligence Paper Is an Embarrassment. Discusses Bostrom’s hilariously absurd argument that we should plow ahead with building god-like ASI even if there’s up to a 97% chance of total annihilation.

9. Sam Altman Can’t Stop Lying. Altman lied about the Molotov cocktail thrown at his house. He lies about his views on AI and democracy. He lies about virtually everything. I provide a number of egregious examples.

   Subscribe now

Human Extinction (Ethics and History):

1. Making Sense of the “Human Extinction” Debate. This provides a comprehensive but highly accessible introduction to the ethics of human extinction. It introduces a crucial ethical distinction between Going Extinct and Being Extinct, disambiguates the surprisingly polysemous term “human extinction,” and delineates the three major positions one could take on the ethics of our extinction: equivalence views, further-loss views, and pro-extinctionist views.

2. A Revolutionary Paradigm Shift, on Par with the Copernican and Darwinian Revolutions, Is Happening Right Now and No One Is Talking About It. Argues that the realization that human extinction is possible, which has slowly unfolded over the past two centuries, constitutes a profound revolution in our thinking about the nature of humanity and our place in the cosmos on par with the Copernican and Darwinian revolutions. For the entirety of Western history — even when some Ancient Greeks believed that demographic extinction is possible (i.e., the disappearance of our species, which could be permanent or temporary) — virtually everyone accepted the fundamental indestructibility of our species. Recognizing the mortality of our species is a massive paradigm shift, which is still ongoing.

3. Human Extinction: A Brief Guided Tour. This was published before the newsletter was officially born. But I’m including it here because it condenses my 200,000-word book into a “mere” 20,000 words. It’s the tl;dr of my tome.

Other Topics:

1. Chomsky’s Connection to Jeffrey Epstein and Other Sexual Predators. I report on Chomsky’s links to Epstein, as well as Lawrence Krauss. One of the most disheartening to write, as Chomsky was (past tense) a moral hero of mine. I also discussed the issue on Briahna Joy Gray’s podcast.

2. Here’s Why SpaceX’s Mission Is Bullshit. Argues that to achieve its mission of extending the “light of consciousness” to Mars and beyond, SpaceX will have to solve one of the following monumental — perhaps even insurmountable — problems: (1) terraform Mars, despite the fact that we can’t even terraform Earth. Or (2) build machines that are conscious, and devise a highly reliable and accurate “test” for determining whether particular machines are in fact conscious. Those are SpaceX’s two options, and neither appears feasible.

3. A Philosopher Explains Quantum Computers. Provides a roughly 9,000-word introduction to quantum mechanics. If you’re looking for a highly accessible introduction to superposition, entanglement, interference, non-locality, qubits, and computational complexity theory, check this out!

4. Pursuing a Good Life in a Bad World. Explains how I maintain a cheerful disposition despite holding perhaps the most pessimistic view of the world possible, namely, that the “world is a living hell, a disaster zone, a waking nightmare, a horror show overflowing with unfathomable amounts of sorrow, anguish, and heartache.” Relatedly:

5. Are You Suffering from Crisis Fatigue? Why yes, yes we are! Examines the psycho-emotional toll of dealing with the polycrisis.

AI Bullshit:

1. Why You Should Never Use AI Under Any Circumstances for Any Reason No Matter What. It’s funny how the articles (and tweets) I’ve spent the least time on sometimes end up becoming the most popular. With 3,608 likes as of this writing, this is by far my most read article. It compiles a number of “AI bloopers,” which people apparently enjoy chuckling at. :-)

2. GPT-5 Should Be Ashamed of Itself. A companion piece to the article above, published a few weeks prior.

Climate Change:

1. Covering the Alps in Giant Tarps and Other Climate-Related News. Provides an alarming snapshot of the climate crisis. It’s one of several articles I’ve written on the topic, though it hasn’t been a primary subject of this newsletter thus far.

   Subscribe now

Questions for Readers:

I’d love to know more about what you like — and don’t like — about this newsletter (please feel free to comment candidly below). Which topics should I cover more? Would you like me to include short commentary videos in my posts? Are my articles, in general, too long or too short? And so on.

Any other thoughts on how I could make this newsletter better? Any suggestions on what I can do to grow my audience? I’m all ears. :-)

As always:

Thanks for reading and I’ll see you on the other side!

I’ve followed his work since about 2005, and it’s been clear to me for some time that Bostrom considers himself to be a towering intellect of cosmic-historical importance. He is not shy when boasting about his supposed accomplishments, even quoting on his website an article that describes him as the “Swedish superbrain.” For many years, he bragged that — to quote his profile on Edge.org — “his performance as an undergraduate set a national record in Sweden,” though he backed away from this after Simon Knutsson pointed out that there is no evidence to support such a claim. (See the “Potentially Dishonest Self-Promotion” section of this article.1)

According to a TESCREAList named Diego Caleiro, as Caleiro departed FHI many years ago, Bostrom turned to him and “made a gesture like Atlas, holding the world above his shoulders, and quasi letting the world fall, then readjusting.” Caleiro writes that “while funny, I also understood the obvious connotation of how tough it must be to carry the weight of the world like that.”

Yet, because I’m so conversant with the foundational literature of the TESCREAL movement, I’ve noticed that a large number of the “original,” “novel,” even (supposedly) “groundbreaking” ideas attributed to Bostrom actually originated with other thinkers. As early as 2018, I was posting about this on social media and privately sharing examples with colleagues — some of whom were rather shocked to see Bostrom’s work juxtaposed with that of others (from whom he borrowed without sufficient attribution). I wrote to one colleague that Bostrom’s “genius” seems to be taking ideas from other theorists and repackaging them in his own language such that they’re just different enough for him to avoid being accused of intellectual plagiarism. His oeuvre reminds me of the line, often misattributed to Picasso: “Good artists copy; great artists steal.”

Here are just a few examples. Don’t take my word for it — decide for yourself:

1. Existential Risk Scenarios

1.1 Bostrom’s 2002 Paper

Perhaps the most “famous” paper written by Bostrom is his 2002 “Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards.” I first read this circa 2005 and was struck by its breadth and originality. Bostrom offers a technical definition of “existential risk,” which denotes any event that would prevent us from creating a (utopian) posthuman civilization. (Hence, the term is intimately bound up with his version of transhumanism.)

About half of the paper is a detailed classification of different types of existential risk scenarios, such as2:

1. Deliberate misuse of nanotechnology and a nanotechnology accident (such as the gray goo scenario).

2. Nuclear holocaust.

3. Misaligned superintelligence.

4. Engineered pandemics.

5. Natural pandemics.

6. Physics disasters, e.g., vacuum decay.

7. Asteroid or comet collisions.

8. Runaway climate change.

9. Fundamentalist religious or ecological movement that halts technological development.

10. An extraterrestrial civilization kills us.

11. “Something unforeseen,” because “it would be foolish to be confident that we have already imagined and anticipated all significant risks.” (Take note of the wording.)

12. Examples of the above include “solar flares, supernovae, black hole explosions or mergers, gamma-ray bursts, galactic center outbursts, supervolcanoes,3 loss of biodiversity, buildup of air pollution, gradual loss of human fertility, and various religious doomsday scenarios.” (Take note of each of these threats.)

13. Resource depletion or ecological destruction.

14. Totalitarianism.

15. Insurmountable technological hurdles (that prevent us from developing advanced technologies).

16. Our simulation gets shut down. (Lolz)

17. Dysgenic pressures, whereby less “intelligent” people outbreed their more “intelligent” peers. (Bostrom is a eugenicist.)

He then discusses the Fermi paradox, anthropics, observation selection effects, and the Doomsday Argument. Taking these in order:

The Fermi paradox (which is neither a paradox nor Fermi’s!) points out that it’s surprising we don’t see evidence of technologically advanced civilizations given the age and size of the universe. Hence, Bostrom writes that

there must be (at least) one Great Filter — an evolutionary step that is extremely improbable — somewhere on the line between Earth-like planet and colonizing-in-detectable-ways civilization. If the Great Filter isn’t in our past, we must fear it in our (near) future. Maybe almost every civilization that develops a certain level of technology causes its own extinction.

Observation selection effects concern the question of “what we should expect to observe given some hypothesis about the distribution of observers in the world.” For example, you might think that a large asteroid striking Earth is improbable because we haven’t witnessed one strike Earth in the past 300,000 years (the period during which our species has existed). But if an asteroid had struck Earth during this time, we wouldn’t be here to talk about it having happened. Hence, we will only ever find ourselves in worlds where asteroids haven’t struck in the recent past. This means that the evidence available to us will be biased in certain ways.

The Doomsday Argument is an a priori argument for why human extinction is more probable than we’d otherwise expect — that is, based on empirical observations. The reasoning is interesting: basically, it’s more likely that we’d find ourselves in a large population at the end of history than in the much smaller population of people long ago, when human history was just beginning. We should thus increase our probability estimates of a secular apocalypse happening soon.

1.2 John Leslie’s 1996 Book

Again, I was extremely impressed with Bostrom’s paper when I first read it. It was original and erudite — even polymathic.

But then I read the introductory chapter of John Leslie’s 1996 book The End of the World and was utterly shocked by how similar Bostrom’s paper is to it. (And, as I’ll explain in a moment, we know that Bostrom was very familiar with Leslie’s book.) For example, although Leslie doesn’t use the term “existential risk,” the central topic of his tome is human extinction — the most obvious type of existential risk. In the introduction, he lists scenarios like the following that could potentially have annihilatory consequences4:

1. A disaster from nanotechnology (including the “gray goo” scenario).

2. Nuclear war.

3. Computer-related disaster (e.g., computers replacing us).

4. Engineered pandemics (i.e., “a disaster from genetic engineering,” including “green scum” and altered microorganisms that could render women infertile).

5. Natural pandemics.

6. Physics disasters, e.g., vacuum decay.5

7. Asteroid or comet collision.

8. Runaway climate change.

9. Religious or philosophical ideologies that prevent humanity from surviving, including Schopenhauerian pessimism, ethical relativism, negative utilitarianism, and “person-affecting” ethical theories.

10. Extraterrestrials kill us.

11. “Something-we-know-not-what,” because “it would be foolish to think we had foreseen all possible natural disasters” and “we cannot possibly imagine every single danger which technological advances might bring with them.” (Sound familiar?)

12. Nearby supernova or black hole merger, volcanic supereruptions, pollution, ozone depletion, etc.

13. Extreme ice age.

14. Breakdown of complex systems.

15. Unwillingness to have children.

16. Biological or chemical warfare.

17. Some other kind of technological disaster.6

These lists are clearly very similar. My impression at the time of first reading Leslie was that the bulk of Bostrom’s paper is little more than a summary of Leslie’s opening chapter. Bostrom even uses similar language. For example, in adding “unknowns” sections at or toward the end of each list (exactly as Leslie does), he writes: “It would be foolish to be confident that we have already imagined and anticipated all significant risks.” Compare that with what Leslie says: “It would be foolish to think we had foreseen all possible natural disasters … [and] every single danger which technological advances might bring with them” (italics added to both quotes).

Even more, Leslie then proceeds to discuss the Fermi paradox, anthropics, observation selection effects, and the Doomsday Argument (which he partly developed) — exactly as Bostrom does, except years before Bostrom wrote his paper.

These similarities aren’t a coincidence: Bostrom wrote his PhD dissertation on the topic of anthropics, and relied on Leslie’s 1996 book — mentioning Leslie’s name 135 times and citing the book throughout. Bostrom even thanks Leslie in the Acknowledgments section.

So far as I can tell, the only thing that’s nontrivially original about Bostrom’s 2002 paper is the existential risk concept, which he introduced to capture an idea that was, as it happens, already being discussed by his online Extropian buddies. Whereas Leslie focuses specifically on human extinction, Bostrom recognized that there are a range of survivable scenarios that would also prevent us from creating a posthuman utopia full of “astronomical” amounts of value. That’s not nothing, but it’s not a lot, either.7

2. Where Are They?

In 2008, Bostrom published an article in the MIT Technology review titled “Where Are They? Why I Hope the Search for Extraterrestrial Life Finds Nothing” — which he presumably got paid $1 to $2 per word for.8 His central thesis is that “I hope that our Mars probes will discover nothing. It would be good news if we find Mars to be completely sterile. Dead rocks and lifeless sands would lift my spirit.” He continues:

Conversely, if we discovered traces of some simple extinct life form – some bacteria, some algae – it would be bad news. If we found fossils of something more advanced, perhaps something looking like the remnants of a trilobite or even the skeleton of a small mammal, it would be very bad news. The more complex the life we found, the more depressing the news of its existence would be. Scientifically interesting, certainly, but a bad omen for the future of the human race.

Why does he say this? Given the Fermi paradox, there must be at least one Great Filter that prevents life from developing far enough to colonize the universe and make its presence known. This Great Filter could be behind us or ahead of us.

For example, it could be that the step from non-life to single-celled living organisms (a process called abiogenesis) is extremely unlikely. If so, then we have gotten very lucky — and the mystery of why we don’t see technologically advanced civilizations flitting about the galaxy, given the age and size of the universe, makes perfectly good sense. Or it could be that the step from single-celled organisms to multicellular lifeforms is extremely improbable, or that the subsequent emergence of big-brained creatures like us is highly unlikely.

Alternatively, it could be that all of these steps are fairly probable — i.e., none constitute a Great Filter. Instead, the universe is teaming with big-brained creatures that, just like us, build industrial civilizations, destroy their environments, invent nuclear weapons, run dangerous physics experiments, etc. and then promptly destroy themselves before having initiated a colonization explosion. If that’s the case, then the Great Filter lies ahead of us, meaning that we’ll soon go extinct due to rogue superintelligence, climate breakdown, nuclear war, etc.

What does this imply? It implies that finding evidence of simple lifeforms in the universe — including on our planetary neighbor, Mars — will shift the probability of the Great Filter being behind us to it being ahead of us. If it turns out that simple lifeforms are common, then the Great Filter probably isn’t the step of abiogenesis. That means that the probability of the Great Filter being in our (near) future rises — which is, of course, very bad news for us! It means we might all die in some kind of extinction event in the coming years or decades.

That’s Bostrom’s ostensibly original point. Throughout the article, on my reading, he repeatedly presents this idea as his own. (Read it for yourself and let me know what you think.) “So this is why I’m hoping,” he concludes, “that our space probes will discover dead rocks and lifeless sands on Mars, on Jupiter’s moon Europa, and everywhere else our astronomers look” (italics added). “It would keep alive the hope for a great future for humanity.”

Yet this isn’t his idea at all. It’s clearly included in the 1998 article titled “The Great Filter — Are We Almost Past It?” by Robin Hanson, which introduced the idea of the Great Filter. Hanson writes:

While all of these stories [about how each evolutionary step leading from non-living matter to space colonization could happen] are at least minimally plausible, our main data point implies that at least one of these plausible stories is wrong — one or more of these steps is much more improbable than it otherwise looks. If it is one of our past steps, such as the development of single-cell life, then we shouldn’t expect to see such independently evolved life anywhere within billions of light years from us. But if it is a step between here and a choice to explode that is very improbable, we should fear for our future. At the very least, our potential would have to be much less than it seems. Optimism (as defined here) regarding our future is directly pitted against optimism regarding the ease of previous evolutionary steps. To the extent those successes were easy, our future failure to explode is almost certain.

In other words, if the steps from non-living matter to single-cells, from single-cells to multicellular beings, and from multicellular beings to big-brained creatures are “easy” — if there is no Great Filter making them highly improbable — then we should fear for our future, because it implies that the Great Filter must be ahead of us. Since finding life on Mars would suggest that there isn’t a Great Filter in our evolutionary past, then suddenly the probability that it lies in our future rises.

I read Bostrom’s article before Hanson’s, and once again I believed that Bostrom was articulating a novel insight. But he wasn’t. Yet Bostrom only once mentions Hanson in the MIT Technology Review paper — not to acknowledge that Hanson made the same point, but to parenthetically attribute the term “Great Filter” to him. Bostrom writes: “I borrow this term from Robin Hanson, an economist at George Mason University.”

Meanwhile, over on Bostrom’s personal website, he posted a slightly different version of the article that includes a footnote that says: “I borrow this term from Robin Hanson’s ‘The Great Filter — Are We Almost Past It?’ … a paper which presents an argument similar to the one expounded here.”

If you were to miss that brief footnote in the version posted on his personal website — the only place where he acknowledges the unoriginality of his thesis — then you’d naturally think Bostrom’s article is making a novel point. Readers of the MIT Technology Review most likely did, as did I when I first read it. Turns out Bostrom was just echoing Hanson’s idea.

And for this, he received somewhere between $4,692 and $9,384 (again, $1 or $2 per word), assuming that MIT Technology Review paid writers the same back in 2008 as it does now. That’s a lot of money for rehashing someone else’s point without even noting, as he does in the personal-website (but not officially published) version, that Hanson “presents an argument similar [nay, identical in essence] to the one expounded here.”

3. Superintelligence

In 2014, Bostrom published a surprise bestseller titled Superintelligence: Paths, Dangers, Strategies. It offered a comprehensive overview of the argument for why ASI could pose a dire existential risk. Indeed, Bostrom contends that the “default outcome” of ASI is “doom,” because figuring out how to “align” ASI with the “values” of humanity (by which he really means the values of the TESCREAL worldview) poses enormous technical and philosophical challenges.

This book was hailed as an important contribution to the study of ASI existential risk. Elon Musk endorsed it, as did Bill Gates. Sam Altman called it “the best thing I’ve seen on this topic. It is well worth a read.”

Yet it turns out that much of the book is little more than a summary of ideas outlined by Bostrom’s TESCREAL colleague Eliezer Yudkowsky (who was himself building upon earlier writers like IJ Good, Verner Vinge, etc.). As Ben Goertzel writes in a review of the book, “nearly all the core ideas of Bostrom’s work appeared previously or concurrently in Yudkowsky’s thinking.” In a section titled “Scratch the Surface of Bostrom, Find Yudkowsky,” he reports that

my first reaction, as I paged with curiosity through Bostrom’s gloomy, erudite tome, was to be struck by the remarkable similarity between the arguments, and the general attitude presented, and the perspective on the future of AI put forth by Eliezer Yudkowsky over the last 15 years or so. Yudkowsky has not (yet) written any single document as thorough and carefully-wrought as Superintelligence. But in mailing list and blog posts and online essays over the years, he has articulated a perspective very close to Bostrom’s, and in many ways possessing more depth and nuance.

Goertzel adds that

with only mild oversimplification, one might say that Bostrom’s Superintelligence is a triumphal work synergizing Bostrom’s communication and analytical ability and Yudkowsky’s creative ideas. Of course, this oversimplification isn’t quite fair to Bostrom, who contributed a lot of detailed ideas to his book as well. But as for the Big Ideas in Superintelligence, it really seems that every one was articulated at some point previously by Eliezer Yudkowsky.

So, Bostrom gets some credit for “a lot of detailed ideas,” but the most significant ideas in the book came from Yudkowsky — pictured below.

4. Instrumental Convergence vs. AI Drives

It’s not just Yudkowsky that Bostrom drew from when writing about ASI existential risk. A central premise of the argument for ASI x-risk is what Bostrom calls the “instrumental convergence thesis.”

Bostrom first delineated this thesis (so far as I know) in his 2012 article, “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents.” It states that agents with a sufficiently high level of “intelligence” will predictably pursue a finite set of intermediary or “instrumental” goals, given a wide range of final goals (i.e., the ultimate goals the ASI wants to achieve). Such instrumental goals include:

• Self-preservation: ensuring that it stays alive, because an agent can’t reach its final goals if it perishes.

• Goal-content integrity: ensuring that it’s goals stay the same over time, because an agent at T1 can’t reach its final goals if it allows those final goals to change at some later time T2.

• Cognitive enhancement: improving its cognitive abilities, including its rationality, as this will help it find new and better ways to attain its final goals.

• Technological perfection: create ever-more powerful technologies, as this will help it bend the world toward realizing its final goals.

• Resource acquisition: acquire as many resources as possible, as resources are necessary for achieving most final goals that it might have.

Whether the final goal of an ASI is curing cancer, calculating the 100-millionth digit of pi, or converting the entire universe into data centers, we can predict — the claim goes — that in trying to achieve such goals, the ASI will attempt to preserve itself, prevent its goals from being altered, improve its cognition, perfect technology, and acquire maximal resources. This is the instrumental convergence thesis.

Once more, I came away from this paper after reading it 10+ years ago thinking that it was yet another publication of notable originality from Bostrom. Only later did I notice a footnote hidden at the bottom of one of the pages that says a guy named “Stephen Omohundro has written two pioneering papers on this topic,” one being “The Basic AI Drives,” published in 2008 — four years before Bostrom’s paper and six years before Superintelligence. Bostrom goes on to say that he doesn’t like Omohundro’s “AI drive” terminology. He thinks “instrumental goals” is better. He then adds at the very end of this footnote that his “treatment here also differs from that of Omohundro in some other more substantial ways, although the underlying idea is the same.”

I would agree that “the underlying idea is the same,” because it obviously is. For example, in Omohundro’s paper, he identifies the following as AI drives (i.e., instrumental goals) that any sufficiently intelligent system will almost certainly pursue:

• “A strong drive toward self-preservation.” Again, if the AI ceases to exist, it won’t be able to achieve its final goals, which means it will take steps to ensure its survival.

• “AIs will try to preserve their utility functions,” which Bostrom calls goal-content integrity.9

• Self-improvement, such as developing “more efficient algorithms, more compressed representations, and better learning techniques” to augment its cognitive abilities. This also means the AI systems will try to make themselves more rational. In other words, cognitive enhancement.10

• “AIs will want to acquire resources and use them efficiently,” given that “all computation and physical action requires the physical resources of space, time, matter, and free energy.”

Bostrom’s list isn’t entirely identical with Omohundro’s. But it’s very close. Yet, when TESCREAL folks talk about the topic, almost no one cites Omohundro. Some aren’t even aware that he published an article strikingly similar to Bostrom’s. Rather — as with all the examples above — Bostrom gets most of the credit, as if he was the first to hit upon these ideas.

5. Letter From Utopia Vs. Letter to Mother Nature

Back in 2008, I emailed one of the founding members of the Extropian movement. (Bostrom was once an Extropian.) In the email, I mentioned Bostrom’s essay “Letter from Utopia,” as I was using it for an art project. The Extropian liked my art project but said:

A suggestion is to read Max More’s “Letter to Mother Nature” [1999] written years before Nick grabbed onto the idea and then did his own version with no credit or mention of Max. … I just wish that you were using the original idea and the original letter because that would give your [work] integrity and honesty.

Bostrom published “Letter from Utopia” in 2008 (though he claims to have first circulated a draft in 2006), while More read his “Letter to Mother Nature” at an Extropianism conference in 1999. Not only are the titles similar, but the content overlaps considerably. For example, More lists seven “amendments” to the “human constitution,” namely:

1. Overcome aging and death.

2. Expand perceptual range (new senses).

3. Enhance cognition.

4. Supplement the neocortex with a “metabrain,” i.e., a “distributed network of sensors, information processors, and intelligence.”

5. Use genetic engineering to control biological and neurological processes.

6. Refine emotions and improve rationality.

7. Control our biochemistry and integrate technology into our bodies.

The goal is for “these amendments to our constitution [to] move us from a human to a posthuman condition as individuals.” Along exactly similar lines, Bostrom’s “letter” is written by a fictional posthuman who declares that we could become posthuman if only we pursue “three fundamental transformations,” namely:

1. Overcome aging and death.

2. Enhance intelligence and perceptual range. (“And then ask yourself, what other harmonies are there in the air, that you lack the ears to hear? What vaults of value are you witlessly debarred from, because you lack the key sensibility? … Your brain must grow beyond the bounds of any genius of humankind, in its special faculties as well as its general intelligence …”)

3. Elevate well-being. (Maximize our ability to experience pleasure. Increase our rationality: “All emotions have a function. Prune and weed carefully lest you accidentally reduce the fertility of your plot.”)

Not once does Bostrom cite More’s earlier “letter” as a source of inspiration. Not once does he indicate that his “letter” is strikingly similar to More’s. Yet almost no one cites More; instead, the citation goes to Bostrom.

6. Astronomical Waste

One of the foundational papers of longtermism — the claim that we should become posthuman, colonize space, and flood the universe with digital beings living in giant computer simulations — is Bostrom’s 2003 paper “Astronomical Waste: The Opportunity Cost of Delayed Technological Development.” In 2022, Elon Musk even retweeted a link to this paper along with the line (in the original tweet): “Likely the most important paper ever written.”

This paper played a huge role in my conversion to the longtermist/TESCREAL worldview. But, once again, I was later shocked to discover that the central idea wasn’t Bostrom’s.

In brief, the paper argues that if we colonize space — especially as digital posthumans — the future population could be enormous. Bostrom estimates some 10^38 digital people within the Virgo Supercluster, a cluster of galaxy clusters that includes our own Milky Way. (That’s a 10 followed by 38 zeros, by the way.)

If one accepts a utilitarian perspective, this means the expected value of ensuring the realization of all those future people (assuming they have net-positive lives) far exceeds the value of, e.g., helping poor people today. If you’re a truly altruistic person who aims to maximize your philanthropic impact, then you should focus not on present-day people but those living perhaps millions or billions of years from now. It’s not that future digital people are inherently more valuable. It’s just a numbers-game: there are so many more of them (potentially) than present-day people, they overwhelm the simplistic equations of utilitarianism’s moral mathematics.

Bostrom’s conclusion is that we should colonize the universe as soon as possible, because every single century, decade, and even second of delaying this results in an “astronomical” loss of potential value — i.e., the non-existence of people who could have otherwise been created in some distant galaxy if only we had started colonizing just a bit sooner. However, Bostrom also concludes that “priority number one, two, three and four should … be to reduce existential risk.” Space colonization should therefore be priority number five once we secure our survival. Perhaps you can see why Musk loves this paper.

In the second paragraph of Bostrom’s paper, he writes that “one recent paper speculates, using loose theoretical considerations based on the rate of increase of entropy, that the loss of potential human lives in our own galactic supercluster is at least ~10^46 per century of delayed colonization.” This cites a paper by Bostrom’s colleague, Milan Ćirković, titled “Cosmological Forecast and its Practical Significance,” published one year before “Astronomical Waste.” Curious, I read Ćirković’s piece and was quite surprised to find that he makes an almost identical argument to that of Bostrom, though the details and framing are different.11 At the risk of going into too much detail, here’s what I write about this paper in my 2024 book:

Ćirković argued that physical eschatology paired with recent developments in anthropics (from both [John] Leslie and Bostrom) carry potentially urgent practical implications for “intelligent observers interested in self-preservation and achieving [the] maximum of its creative potential.” Specifically, the timing of space colonization could make a significant difference to the amount of resources available to advanced civilizations, given our best current understanding of our “cosmological situation.”

That’s exactly what Bostrom argues. Even more:

As [Ćirković] expressed the idea, decision-making performed today, as far as humanity is concerned, may have enormous consequences on very long timescales. In particular, an overly conservative approach to space colonization and technologization, may result (and in fact might have already resulted) in the loss of [a] substantial fraction of all possible observer-moments humanity could have had achieved. The term “observer-moment” comes from Bostrom, who defined it as “a brief time-segment of an observer.” Thinking in terms of observer-moments instead of observers is relevant to anthropics — that is, to reasoning about one’s location in space and time based solely on the fact that one is an intelligent observer. But it may also be relevant to estimating our “potential” over time: since “different observers may live differently long lives, be awake different amounts [of] time, … etc.,” counting observer-moments rather than observers can give a more accurate measure of the bigness or value of the future. There could, after all, be a large number of observer-moments even if there are relatively few observers, and vice versa.

The point is that Ćirković used this concept to argue that we should develop advanced technologies and colonize the universe as soon as possible, given that every passing moment our cosmic endowment of negentropy is going to waste as stars burn up their limited reservoirs of hydrogen. Consequently, by failing to advance technology and colonize space, we could lose — and maybe have already lost — a “substantial fraction of all possible observer-moments [that] humanity could have … achieved.” To underline the time-sensitivity of this situation, Ćirković calculated that if a “future hypercivilization” could extract all the energy output of the stars populating the Virgo Supercluster, then “the number of potentially viable human lifetimes lost per century of postponing … the onset of galactic colonization is … 5 × ~10^46.” Thus, assuming some correlation between the number of lifetimes or observer-moments and the fulfillment of our “creative potential,” this provides a very strong prima facie reason against “an overly conservative approach to space colonization and technologization.” (Italics and a paragraph break were added to the original text.)

What’s different between the central arguments and conclusions of Bostrom and Ćirković? Not a ton. Both calculate the costs of postponing space colonization. Both provide estimates of how many people there could be in the Virgo Supercluster. Both argue that we should therefore accelerate the development of technology and colonize space as soon as possible. Bostrom says that the loss of value with every century of delayed colonization “boggles the mind,” while Ćirković writes in similar language that the loss “per a century delay in starting the colonization is astonishing by any standard.”

The biggest difference is that Bostrom frames the issue in terms of “existential risk” (a word that Ćirković doesn’t use) and provides a more explicit ethical foundation — utilitarianism — to support the conclusion he shares with Ćirković. For years, I was under the impression that Bostrom was doing a lot more in his paper than recapitulating Ćirković’s idea of calculating how many future people there could be and then arguing that we should colonize space ASAP to prevent “astronomical waste.”

7. Paperclip Maximizer

If you look up “paperclip maximizer” on Wikipedia, you’ll find a section that opens with this: “The paperclip maximizer is a thought experiment described by Swedish philosopher Nick Bostrom in 2003.” But this is misleading, as it suggests that Bostrom is its progenitor. Indeed, while the idea is regularly attributed to him, it appears that Yudkowsky likely first came up with it.

In a 2018 interview, Sam Harris asks Yudkowsky: “Does this thought experiment originate with Bostrom, or did he take it from somebody else?” Yudkowsky replies: “As far as I know, it’s me.” He adds that it might still be Bostrom’s idea, though he was “the first one to say ‘paperclips’” if the idea originated on mailing lists that he was a part of. (No one, to my knowledge, has double-checked this.)

It’s fine that Yudkowsky doesn’t much care about attribution, but contrast his openness about this issue to Bostrom’s. I have never once seen Bostrom suggest in any interview that he might not have invented the thought experiment. (Let me know if I’m missing something.) He never seems to skip an opportunity to take credit for an idea attributed to him, whether or not that idea is actually his.

(For some useful information that calls into question this example, see this comment from Hypnosifl below.)

Similarly:

8. Pascal’s Mugging

This refers to a thought experiment that shows how maximizing expected value (or utility) — which some would say is integral to rationality — can lead one to make decisions that appear irrational. Expected value is (roughly speaking) what you get when you multiply the probability of an outcome by its value. For example, an outcome with a value of 10 and a probability of .5 has the same expected value as an outcome with a value of 1,000 but a probability of 0.005. (That’s because 10 x .5 = 5, and 1,000 x 0.005 = 5.)

Pascal’s mugging results when enormous values are paired with tiny probabilities. Mitigating speculative existential risks provides an example: if there could be 10^58 happy digital people in the future, as Bostrom calculates, then it might be worth spending huge amounts of money mitigating a highly speculative existential risk scenario even if one judges the probability of that scenario to be tiny. That is to say, multiplying the payoff (value) of the outcome (10^58 happy people) by the (low) probability could still yield a very high expected value.

In 2009, Bostrom published a paper on Pascal’s Mugging in the prestigious philosophy journal Analysis. But the idea was first introduced by Yudkowsky in a 2007 blog post titled “Pascal’s Mugging.” Bostrom briefly gestures in the direction of this fact in a footnote, saying: “Related scenarios have recently been discussed informally among various people. Eliezer Yudkowsky named the problem ‘Pascal’s mugging’ in a post on the Overcoming Bias blog.” Yet it was Bostrom’s name on the Analysis paper, which he was then able to add to his already “impressive” CV, thus boosting his academic “credibility.”

In Yudkowsky’s version, a mugger says to their intended victim: “Give me five dollars, or I’ll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills [an unfathomably large number of] people” (italics added). Even if one deems the mugger’s claim to be extremely improbable, this low probability is offset by the hugeness of the number of people who would be killed. Hence, expected value theory implies that one should hand over the money.

In Bostrom’s version, the mugger says: “I have magical powers. I can give you any finite amount of money that you might ask for tonight. What’s more, I can give you any finite amount of Utility that I choose to promise you tonight” (italics added). Rather than threatening to simulate and kill people, the mugger offers to give the victim a phenomenal amount of money tomorrow if they hand over their wallet right now.

It’s fairly well known that Yudkowsky originally devised the scenario, and that Bostrom basically just re-worded and published it a highly reputable philosophy journal. However, I’d consider it rather odd — and problematic in an ethical sense — if a friend of mine wrote a blog post about an idea, and then I took that idea, changed some minor details, and managed to get a major publication out of it. What do you think?

Subscribe now

A Project of Self-Aggrandizement

There are many more examples that could be adduced.12 The point isn’t to shame Bostrom for drawing from other scholars. Every academic does this, of course — it’s the way intellectual progress proceeds. I stand on the shoulders of the giants who came before me, and the next generation will stand on my shoulders by building upon whatever newish ideas (if any) I might have been lucky enough to stumble upon in my brief time here on Earth.

However, in most cases, academics are pretty good at properly attributing the origins of ideas they further develop or elaborate. I do not see this in Bostrom’s works at all. My sense, from reading him since ~2005, is that he goes out of his way to hide the fact that many of his “best” ideas originated in the work of his colleagues. Nowhere in his writings does he say things like the following:

• “Reproducing much of John Leslie’s taxonomy of extinction hazards, I will enumerate various scenarios that give rise to what I call ‘existential risk.’”

• “As Robin Hanson alluded to, it would be very bad if we were to discover life elsewhere in the universe, given the Great Filter framework.”

• “Drawing heavily from the unpublished work of Eliezer Yudkowsky, I will outline an argument for why ASI could bring about the annihilation of humanity.”

• “Developing an idea introduced by Stephen Omohundro, I will examine what I call ‘instrumental goals’ that ASI will naturally pursue.”

• “Echoing Max More, this ‘letter’ outlines various projects we should pursue in order to become posthuman.”

• “I will argue, as Milan Ćirković does, that every century of delayed space colonization results in an astronomical loss of human potential/value.”

• Etc.

The vast majority of scholars would have included such remarks if they had written Bostrom’s papers. It’s standard to not take credit for other people’s intellectual labor; it’s common to be generous in citing others. (Plenty of my own scholarship draws from other people, sometimes heavily. But when I do this, I make it clear that I’m relying on their insights. Nothing special here — precisely because it’s standard practice!)

Yes, Bostrom does cite the people above, but not in a way that properly conveys the extent to which his work takes from them — or so I would argue. Pair this with the large volume of evidence that Bostrom thinks of himself as a super-genius of almost messianic significance — he’s saving not just the world, but the entire future of posthumanity — and it starts to look like these weren’t exactly innocent mistakes.

From my perspective, Bostrom is at best a second-rate thinker whose real talent is repackaging other people’s insights in accessible-yet-academic language that appeals to a wide audience. This is what he’s good at — and we should give him credit for that. As Goertzel notes, the guy “is a very good writer, mixing academic rigor with occasional color and humor in a classic European style.”13 But for originality, creativity, and insight, Bostrom gets a D- in my gradebook.

One can see this — or at least I do — as part of a life-long project of self-aggrandizement. And it’s been largely successful: Bostrom has become “one of the most-cited philosophers in the world,” as he boasts on his website. This is, however, thanks to the success of publications like those mentioned above: Superintelligence (12,152 citations on Google Scholar), “Existential Risk” (1,392 citations), “The Superintelligent Will” (674 citations), “Astronomical Waste” (402 citations), “Letter from Utopia” (219 citations), “Pascal’s Mugging” (104 citations), and so on. On his website, Bostrom adds that his

writings have been translated into more than 30 languages; he is a repeat main-stage TED speaker; he has been on Foreign Policy’s Top 100 Global Thinkers list twice and was included in Prospect’s World Thinkers list, the youngest person in the top 15.

This all looks rather less impressive when you realize that the reason he has so many “accomplishments” is because of publications that drew heavily, often without sufficient attribution (in my opinion), from others.

***

While writing this penultimate section, I decided to look into Bostrom’s boast that he’s among the most-cited contemporary philosophers. When Gemini answered my Google query about this, it told me the following:

The claim that Nick Bostrom was the “most-cited professional philosopher under the age of 50” comes directly from his official professional biography used across academic institutions, speaker bureaus, and publishers.14

Oh, so the biography that he wrote is the citation for a claim in his biography! After more digging, I found this ranking from after 2020, which lists him as the 58th most-cited philosopher at the time. Two thoughts:

1. Does 58 really count as “one of the most-cited philosophers in the world”? I’d have thought this would be reserved for the top 10, or perhaps the top 50. Bostrom is neither in the top 10 nor top 50.

2. I have never once seen any other philosopher above Bostrom on the list bragging about their ranking. (Have you?) Consider that David Chalmers — who is lionized within philosophy — is listed as #10 in the rankings. Yet nowhere on his website does he mention this. (According to Google Scholar, Chalmers has nearly twice as many citations as Bostrom.) The same goes for Timothy Williamson at Oxford, listed as #8. Even Martha Nussbaum, who’s the most-cited living philosopher, says nothing about it on her website or CV. Why? Because, while academics are expected to proudly list their awards and publications, overt self-promotion is often seen as bad form. It’s cringe-inducing and doesn’t reflect well on the humility of the individual.

But Bostrom, once again, never misses an opportunity to exaggerate and promote himself. It’s what his life-long project of self-aggrandizement is all about.

Here’s another example:

On his personal website, Bostrom boasts that his book Deep Utopia, which was published with IdeaPress Publishing, a pay-to-publish company, has won a whole bunch of awards. As of August 7, 2026, he writes:

Chinese translation of Deep Utopia is now out. Also the second print run in English and the Audiobook. The book has received still more awards.

Such awards include:

But if you look into these awards, most of them are trivial — you might even say bullshit — “awards” given to dozens and dozens of books. For example, over 70 books received the “2024 PenCraft Book Award.” In the same “fiction” category that includes Bostrom’s book (even though his book is supposed to be nonfiction!), one finds titles like these:

And:

Another “award” that Bostrom’s book won is the “Living Now Book Awards.” It’s official website says:

The Living Now Book Awards feature over 30 categories, celebrating exceptional lifestyle, homestyle, personal development, wellness, and Christian-themed books

Another website, independentpublisher.com, describes the award as follows:

The Living Now Book Awards celebrate the innovation and creativity of newly published books that enhance the quality of our lives, from cooking and fitness to relationships and mature living.

For an academic, this is an embarrassment.

Moving on to the “Independent Press Awards” that Bostrom also boasts of having won, this is what you’ll see if you click on their website. Amid dozens and dozens of books, you’ll see Bostrom’s about halfway through:

The obvious interpretation of Bostrom’s website (screenshot above) is that his book was singled-out as an outstanding work of literature by various award-granting groups. But this is completely misleading: these are, again, bullshit awards given to tons of books.

Yet Bostrom can’t resist mentioning them because — like Donald Trump — he is an insecure narcissist who’s obsessed with exaggerating his accomplishments for the purposes of self-aggrandizement. Trump has his fake TIME magazine cover; Bostrom has these awards, his fabricated academic “record” in Sweden, his thousands of citations for papers containing unoriginal ideas, etc. etc.

Why Has No One Said Anything?

An interesting question is why the people from whom Bostrom has borrowed so heavily without proper attribution have been okay with this. Literally no one, to my knowledge, has complained about intellectual plagiarism. Why not? I brought this up to a former researcher at Bostrom’s Future of Humanity Institute (FHI), who responded as follows:

My view, for what it’s worth, is that Bostrom has played fast and loose with the ideas of people who were happy to be plagiarized because they thought “he’s going to be a big Oxford philosopher and having someone like that in our pocket will be good for us and the things we care about so let’s all just go along with it.” I.e. I think maybe you [Émile] are not being hard enough on the people who have let Bostrom plagiarize them without protest, because they only did it because it was in their interests to do so, they weren’t just being charitable to a sub-genius with an ego trip!

On this view, personal self-interest is why no one has spoken out. That seems plausible to me. However, the fact that others are okay with Bostrom taking their ideas doesn’t make it right. Consider a friend of mine who writes an excellent philosophy paper and then says to me (in some parallel universe): “I know you need another publication for tenure, so feel free to put your name on this one and publish it as yours. We’ll never mention this to anyone.” If I were to publish their paper as mine, I would still be committing academic fraud — even if I had their full permission. Standards don’t dissolve if consent is granted.

So, it’s a bit frustrating that no one has said anything. If the people above were to publicly complain, then (a) they’d have a very good case for being upset, and (b) the result could well be a significant scandal for Bostrom. As far as I know, this article is the first to lay out the evidence and make that case. Perhaps it will inspire someone to speak out, especially now that Bostrom is no longer at Oxford.15 We’ll see.

Subscribe now

Have you come across important examples that I’ve missed? Please do share them below (or feel free to email me). As always:

Thanks for reading and I’ll see you on the other side!

ADDITIONAL CONCERNS (added August 6, 2026)

In Deep Utopia, Bostrom explicitly thanks ChatGPT-4 and Claude 2 “for extensive comments on the manuscript.” Curious, I ran the abstract of his recent paper “Optimal Timing for Superintelligence” (which argues that we should build ASI even if there’s up to a 97% chance of everyone on Earth dying) through Pangram and another AI detection program. This was the result:

Pangram is rarely wrong when it comes to flagging something as being AI-generated, so it’s safe to assume that Bostrom did indeed use AI to write the abstract. I haven’t checked the rest of his paper, or his Deep Utopia, because I don’t have the money to pay for Pangram — but if others are inclined to do this, please let me know what the results are.

Also, user Hypnosifl added some additional examples in a comment below, which I am reproducing here in full:

Bostrom’s anthropic arguments all revolve around what he called the “self-sampling assumption” (SSA), so also might be worth noting that although he coined the term, John Leslie’s 1992 paper “Time and the anthropic principle” (doi is 10.1093/mind/101.403.521 if you want to look it up on sci-hub) lays out the basic idea on p. 523:

“Observers can most expect to find themselves in the spatiotemporal regions containing most of them.

“Again, suppose that almost all intelligent life is based on water and exists on planets when many stars still shine. ... It should then come as no surprise to us that we are on a watery planet and see a starry sky.

“Compare the case of geographical position. You develop amnesia in a windowless room. Where should you think yourself more likely to be: in Little Binding with a tiny population, or in London? Suppose you remember that Little Binding’s population is fifty while London’s is ten million, and suppose you had nothing but those figures to guide you. (You did not recall ever having been in either place; you had no theory that London fogs induce amnesia; and so on.) Then you should think that you are in London. Suppose, by contrast, you see no grounds to prefer the belief that you are in the larger of the two places. Forced to bet on the one or on the other, you bet you are in Little Binding. If ten per cent of the people in the two places developed amnesia and betted as you had done, then there would be one million losers and only five winners. So, it would seem, betting on London is more rational. The right estimate of your chances of being there rather than in Little Binding, on the evidence available to you, could well be reckoned as ten million to fifty.

“What if the amnesia gives you doubts about whether London’s population is the larger? Finding yourself in London should reduce the doubts: with the larger population, London would be prima facie where you would be more likely to be. Again, if London and Little Binding were the only places you could be, then finding yourself in Little Binding could suggest strongly that London was a fiction, if its fictitiousness had no great initial improbability.”

Leslie attributes the idea to Brandon Carter (the astrophysicist who coined the term ‘anthropic principle’) who was implicitly using this form of anthropic reasoning when he formulated the doomsday argument. Leslie also used the language of “sampling” in his 1997 paper “Observer-relative chances and the Doomsday Argument” (doi 10.1080/00201749708602461), writing on p. 429 of an example with a hundred individuals where ‘In effect, each can treat herself as a sample drawn at random from the hundred’, and that an external observer lacks the kind of first-person information that the hundred subjects of the experiment have, writing that this is ‘information which can be treated as the equivalent of having sampled the hundred randomly’. P. 436 brings up Eckhardt’s term for this type of reasoning, the ‘human randomness assumption’ (from Eckhardt’s 1997 paper ‘A Shooting-Room View of Doomsday’, doi 10.2307/2564582, which defines the assumption on p. 248 as ‘We can validly consider our birth rank as generated by random or equiprobable sampling from the collection of all persons who ever live’), but Leslie brings up his own view that we can’t use this type of reasoning ‘if the world is importantly indeterministic’ (I think his argument here also depends on what philosophers call the ‘A-theory of time’ with an objective present and a potentially open future, as opposed to the ‘B-theory’ where future facts are just as fixed as past ones and the ‘present’ is purely an observer-relative concept like ‘here’ is for space).

Bostrom I think claims too much credit for the originality of the SSA in his 2002 book Anthropic Bias--when he first introduces it at https://anthropic-principle.com/q=book/chapter_3/#3d he claims that Carter and Leslie only discussed the “weak” and “strong” anthropic principles which say that in a multiverse you’ll find yourself in one of the universes that allows intelligent life, but do not say you are more likely to be in a universe that contains more observers. And in chapter 6 at https://anthropic-principle.com/q=book/chapter_6/ he says Leslie discussed only a ‘vague’ approximation to the idea: ‘Leslie talks of the principle that, lacking evidence to the contrary, one should think of one’s position as “fairly typical rather than highly untypical”. SSA can be viewed as an explication of this rather vague idea.’ His only mention of Eckhardt is footnote 2 at https://anthropic-principle.com/q=book/chapter_7/ which cites him as one of many who have objected to the Doomsday Argument, but he does not mention Eckhardt’s definition of the ‘human randomness assumption’ which is just the SSA applied to human history.

Bostrom did give both Leslie and Eckhardt more credit for ideas basically equivalent to the SSA in his 2000 paper “Observer-Relative Chances in Anthropic Reasoning?” (doi 10.1023/A:1005551304409) which to me suggests the exaggeration of the originality of the SSA in his book is dishonest, or at best sloppy scholarship. On the first page of this paper Bostrom cites Leslie’s 1997 paper, and says ‘In a recent paper,1 John Leslie argues that a version of the weak anthropic principle gives rise to paradoxical kind of observer-relative chances ... The anthropic assumption that is used in deriving these observer-relative chances is that: Any observer should regard herself as a random sample from (some suitable subset) of the set of all observers. We can call it the self-sampling assumption.2’ And footnote 2 at the end of the sentence also cites Eckhardt’s ‘human randomness assumption’ by name, though he criticizes this formulation because it only applies to humans and not other forms of intelligence. (But he does note in that footnote that Carter’s version of anthropic reasoning did not have this restriction, and I’d also point to the quote about most aliens being based on water from Leslie’s 1992 paper above, so this nitpick about Eckhardt’s version is not enough to claim his own SSA as an original idea.)

14

It adds: “Because citation metrics by age are not tracked by a single official body, this description is an aggregated data point originating from Google Scholar rankings for analytic philosophers.” (I’ll compare Bostrom’s Google Scholar citations to David Chalmers’ in a moment.) Here’s a screenshot:

Hmm.

living through this kind of eclipse of the human intellect, where we’re in the final days of humans being the primary actors on this planet. And that soon, machines will rise. There is this irony in that I think that whole transformation, I think humans … will actually go through a very “main-character energy” period of time, as that transformation occurs, even if it ultimately does mean that the machines ultimately become the primary actors, there’ll be this period.

He claims that he doesn’t “necessarily believe this,” yet I don’t get the impression that he’s particularly bothered by what he describes as “this final flowering of humanity in the birthing of the machine intelligence.”

Anyway, he went viral over the past week for a rather controversial tweet about the repercussions of Kimi K3, a powerful new AI model — in fact, the largest open-source AI in the world right now — developed by the Beijing-based startup Moonshot.

This model rivals the best models released by AI companies in the US like Anthropic and OpenAI. As ABC reports:

K3’s release has also put pressure on stocks of American technology titans on worries that more affordable Chinese AI models may undercut the pricing power and demand at the U.S. AI companies, even as U.S.-led restrictions have already barred China from accessing some of the world’s advanced technologies, including the most cutting-edge chips.

***

Enter Mr. Ball, who said on X that Kimi K3 is “a very good model,” and “its performance [can’t] be explained away by distillation or anything like that.”

Ball continued:

One probable outcome of an open-weight-model-dominant world is full AI communism, which is precisely what China proposes: rather than a market product, AI is a “public good” which will ultimately be provided by the state as a kind of “digital public infrastructure.” This future strikes me as a dystopian hellscape, but I’ve never met an open-weight models advocate who doesn’t ultimately concede this is where things end.

Oh no, “AI communism”! AI becoming a “public good” provided to everyone by the government — as opposed to companies like OpenAI maintaining a stranglehold on the technology. Which is more of a “dystopian hellscape”?

Ball adds that he’s surprised China is okay with open-source AI models, given the “potential risks,” though he suspects that “the reason they are is 75% explained by strategic blindness/lack of AGI-pilledness (the CCP is very Yann Lecun-y in its views of AI).”

Rather, China seems to have the far more level-headed view that generative AI, powered by LLMs, is merely a “tool” that people can use for particular tasks. That’s why they don’t appear especially worried about open-source AI models that are “freely available to use, study, modify, and share.” They aren’t concerned that someone is going to alter these systems and attain AGI, thus sparking the techno-rapture (i.e., Singularity).

***

As you might imagine, many people were highly critical of Ball’s remarks, often mockingly so. For example:

David Sacks — angel investor in Facebook, SpaceX, and Palantir; former COO of PayPal; and Special Advisor for AI and Crypto for the Trump administration — lambasted Ball’s suggestion that “the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models.” (Meaning that businesses could lose out if they use a Chinese model that, say, has backdoors allowing the Chinese government to interfere — or something like that. If the US government alters regulations that punish businesses with such vulnerabilities, those businesses will suffer. This is the threat of regulatory risk.)

In response, Sacks accused Ball of potentially “confessing to a regulatory capture strategy” that would ultimately benefit Ball’s company, OpenAI, by pressuring US companies to eschew much cheaper Chinese models. He writes:

Still others pointed out that “communism” — currently the Trump’s new favorite term-of-abuse for anyone left of center — was also once used to denigrate Linux, an open-source operating system:

Even more, Emil Michael — Trump’s selection for Under Secretary of Defense for Research and Engineering — posted a tweet in which he literally calls Ball a “supreme village idiot” and says that Ball “has perhaps the biggest gap between actual IQ and his own perceived IQ of anyone in the industry (about 40 points).” He writes:

Oh my god. I share all of this mainly for the amusement value. I have never been impressed with Ball (though apparently, to my surprise, Taylor Lorenz has), and now he’s really stepped in it. The fiasco — especially the fact that a major Trump official weighed in to insult Ball — certainly doesn’t help OpenAI, a company that’s been struggling given the success of Anthropic’s Fable and the endless bad press surrounding its “sociopathic” leader, Sam Altman. It will be interesting to see if Ball still has a job in a couple of weeks.

***

For another chuckle, here’s a tweet from someone else I’ve written about:

Camille Paglia is a climate denier who supports NAMBLA (!!), and if you’ve ever read Nick Land’s work (as I regrettably have) you’ll know that it’s a mess of mostly incoherent drivel dressed up in grandiloquent language. So, roon seems to be saying that the new AI models are terrible at thinking and writing. I agree.

***

Bad news on the climate front continues to flood social media like a torrential downpour. A close acquaintance of mine in Virginia recently sent me pictures taken around their house during a Code Purple air-quality warning, meaning that the air is “very unhealthy to breathe in.” That’s worse than a Code Red, but not as bad as a Code Maroon — the most extreme category whereby simply breathing is “hazardous” to everyone, resulting in “emergency conditions.”

This person also told me that they’ve had to endure 108F (42C) temperatures, with humidity sometimes reaching 100%. That means their windows are literally covered in water despite it not raining, as seen below. Absolutely wild stuff:

The air-quality issue is, of course, due to the massive boreal fires in Canada. Some of the footage of the fires is truly terrifying to watch — the huge wall of fire shown in clips below elicits a deep, primordial fear in me, along the lines of videos of large tsunamis and mile-wide tornadoes. Gaia is pissed off at humanity and letting us know this in the most violent ways.

Here in the Alps, the heatwave has subsided but the fields remain brown and the mountains mostly bereft of snow. My CO2 detector indicates that there are 432 ppm (parts per million) of CO2 in the ambient air — to put this in perspective, the Pleistocene (a geological epoch before the Holocene began some 12,000 years ago) contained between 183 and 292 ppm of CO2. This is when our Homo ancestors first emerged in the grassy savanna of East Africa. The last time Earth’s atmosphere contained more than 400 ppm was over 3 million years ago, in the Pliocene, when creatures like this first evolved:

Of note is that some studies suggest that higher amounts of CO2 in the air results in significant — albeit reversible — cognitive deficits, meaning that, as a Yale Climate Connections article put it, climate change is literally making us “dumber.” (And at exactly the moment when we need our brains most to overcome the monumental challenges of the polycrisis!) A more recent discussion of this unfortunate phenomenon can be found here.

Do you have photos or videos showing how climate change has impacted your region of the world? Please feel free to share them below!

***

Finally, some of you might know that I cohost a monthly get-together called the Ethics of Human Extinction Reading Group (with med student Devin Kellis). As the name suggests, we explore the ethical and evaluative implications of our extinction: would the disappearance of humanity be good or bad, better or worse, right or wrong (to bring about)? Why exactly? Would the prospect of extinction undermine key sources of value in our lives today? Do we have obligations to past people to continue to exist? And so on.

As I wrote three years ago in Aeon, even if one doesn’t think that human extinction is likely this century — and I’m one of those who believes the probability is (with qualifications) extremely low, though I also think the likelihood of civilizational collapse (without extinction) is almost certain — the topic nonetheless foregrounds some of the most important issues relating to value, ethics, and meaning.

I mention this because next Saturday at 11:30am Eastern Time, a fantastic scholar at the University of Cambridge, SJ Beard, will be discussing their recent book Existential Hope, a description of which can be found here. I’ve known SJ for about 10 years and never once left a conversation with them without having learned something. They are very insightful, and I thought their book was fantastic. (Alongside Goliath’s Curse, written by SJ’s colleague Luke Kemp, I consider these books essential reading for anyone interested in global catastrophic risks.)

You are absolutely welcome to join us! Just email me at philosophytorres@gmail.com for a Zoom link, as I’m anxious about posting it here due to the ever-present threat of Zoombombers. It should be really fun — or, well, “fun” in scare quotes!

I’m also happy to say that two interviews I gave to large podcasts over the past few years have been selected for inclusion in two different books. And I was just interviewed — yesterday — by the Australian Broadcasting Corporation, which is my third or fourth appearance on Radio National. I’ll keep you posted about when that comes out.

Subscribe now

Wishing everyone the very best and, as always:

Thanks for reading and I’ll see you on the other side!