AI Bloopers / Tiku Interview / Vox on Pro-Extinctionism

(3,100 words)

Despite the Singularity being imminent, AI still struggles with simple tasks like drawing an accurate map of the US. And despite being told that humanoid robots are about to take over factories and serve as our personal butlers (becoming a $38 billion market by 2035), most robots are still pretty lame.

Consider this hilarious video of a humanoid robot being dragged across the stage after awkwardly tripping over some stairs while dancing to Michael Jackson’s “Billie Jean.” Notice how the timing of the synths couldn’t be more perfect — as if to mock the syncopal robot at exactly the moment it passes out:

Meanwhile, someone captured a segment of an AI slop video about the WWE. There are, apparently, a bunch of YouTube channels that publish long-form videos narrated by AI (discussed around 3:00 here). The enslopification of the Internet accelerates! In this case, the AI begins to self-destruct like a washing machine with a brick in it after trying to pronounce “WWE.” I have a facetious ranking of funniness in terms of chuckles and guffaws, where 10 chuckles equals one guffaw. This one gets a full 2.5 guffaws — meaning I doubt you can watch it without laughing out loud:

I don’t think this is fake. I encountered something similar (but not as extreme) when watching a documentary that I thought was narrated by a human. When the narrator tried to pronounce an acronym full of consonants, it produced a bizarre sound that no human would ever make. (I can’t recall what the acronym was — maybe WWF, for World Wildlife fund?). On another occasion, I was watching a different documentary — again, foolishly thinking I was hearing a human — in which the narrator said, “And this cost the US dollar sign 300 million.” God damnit!

As Ronny Chieng put it during a graduation speech at Harvard: “Fuck AI.” I really wonder if, in a very profound sense, we aren’t witnessing an end to the human era — the era in which humans have been dominant. What a sad way for it to end, if that’s the case.

WaPo’s Nitasha Tiku on TESCREAL Doomerism

Taylor Lorenz has a new video in which she interviews journalist Nitasha Tiku of the Washington Post. They discuss how billionaires are moving mountains of money into social media campaigns to raise public awareness of the “existential risks” posed by ASI (artificial superintelligence). In some cases, the very same tech billionaires who invested in the AI companies are now supporting “anti-AI” activists and influencers. I have some thoughts on this interview:

The first part of the video provides a nice — albeit brief — overview of the TESCREAL ideologies. (The heart of this bundle is a techno-utopian vision in which our “posthuman” successors colonize the universe and build a sprawling multi-galactic civilization full of trillions and trillions of “digital people.”) Around 1:41, Tiku says:

So, AI safety is ... an evolving term. Like, this is sort of the one that the community has landed on. This is almost, like, a 25-year-old movement. So if you’ve heard about, like, the singularity, effective altruism, rationalist, transhumanist … it’s kind of at this intersection.

She later mentions longtermism, thus foregrounding 5 of the 7 ideologies in the TESCREAL acronym — the other two being cosmism and Extropianism. The TESCREAL acronym picks out a cultural movement built around libertarian transhumanism — i.e., Extropianism — that extends from the early 1990s up to the present. You can understand it as a more technophilic and utopian iteration of the Californian ideology. All of the major AI companies — DeepMind, OpenAI, Anthropic, and xAI (which is now SpaceXAI) — emerged directly out of the TESCREAL movement. The whole reason the CEOs and cofounders became interested in ASI in the first place was because the TESCREAL worldview identifies ASI as the ultimate means for realizing a posthuman paradise among the stars.

The field of “AI safety” was birthed by the TESCREAL movement. Why? Because some TESCREALists came to believe that there are two fundamentally different kinds of ASI: aligned and misaligned. If we were to build an ASI that’s “aligned” with the TESCREAL worldview, then we’d get the cosmic utopia of posthumanity mentioned above. But if it were misaligned, it will kill everyone on Earth and, astronomically worse, erase our “glorious transhumanist future” (quoting Eliezer Yudkowsky) before we have a chance to start drawing it.

Furthermore, these same TESCREALists — dubbed “doomers” — argue that figuring out how to build an aligned ASI is far more difficult than one would initially assume. So, AI safety was founded to find a solution to the “alignment problem.”

Tiku is absolutely right that the field emerged roughly 25 years ago, in the early 2000s, thanks largely to the work of Yudkowsky. Right now, many AI safety advocates believe that we’re very close to building an ASI, but nowhere close to solving the alignment problem. That means that total annihilation could happen in the very near future.

You can think of this in terms of a race between AI safety research and AI capabilities research. The finish line of the former is solving alignment. The finish line of the latter is actually building an ASI. If capabilities researchers get to their finish line before safety researchers reach theirs, then we’ll all be dead and the transhumanist future will be permanently canceled. When Yudkowsky says, “if anyone builds it, everyone dies,” he’s specifically talking about misaligned ASI — not aligned ASI.

***

At 4:37, Tiku says that AI safety

started with people who are not AI researchers. A lot of them are former philosophers from Oxford — self-taught people who were prescient … about the general direction that AI technology was moving forward. People now give them a lot of props for anticipating some of the concerns that we’ve seen around, like, ChatGPT and very human-like AI, and what this can do to human interaction — how quickly the technology is developing. [This] used to be called, like, concerns about the Singularity …

Here I would push back against Tiku’s characterization of early TESCREALists as being “prescient.” I think she gets things exactly reversed: it wasn’t that these people were prescient, it’s that they actively strove to fulfill their own prophecies — and they succeeded.

Yudkowsky himself played an integral role in launching the ASI race (as I discuss at length in my forthcoming book). He inspired people like Shane Legg, the cofounder of DeepMind, and introduced another DeepMind cofounder, Demis Hassabis, to Peter Thiel after the 2010 Singularity Summit. Thiel then invested in DeepMind, which is how it started.

Sam Altman was also inspired by Yudkowsky, explicitly saying that Yudkowsky is the reason that many people, including himself, became interested in trying to build god-like AI.

Sam Altman@sama

eliezer has IMO done more to accelerate AGI than anyone else. certainly he got many of us interested in AGI, helped deepmind get funded at a time when AGI was extremely outside the overton window, was critical in the decision to start openai, etc.

9:28 PM · Feb 3, 2023 · 1.3M Views

---

122 Replies · 115 Reposts · 2.31K Likes

A New York Times profile describes Altman as a “product” of the EA and Rationalist communities. Anthropic CEO Dario Amodei was an EA-longtermist who hired a bunch of EA-longtermists to work as his company, including Holden Karnofsky, Joe Carlsmith, and Amanda Askell (who was married to the cofounder of EA).

Elon Musk has explicitly said that longtermism is “a close match for my philosophy.” Musk attended numerous conferences before the ASI race took off in which leading TESCREALists like Yudkowsky and Nick Bostrom were present.

And we haven’t even touched on all the money that helped start the ASI race from TESCREAL organizations like Open Philanthropy (now Coefficient Giving) and donors like Jaan Tallinn. (Tiku acknowledges some of this later on.)

It was because of the TESCREAL movement that ASI became widely recognized as the key to creating utopia. TESCREALists popularized the idea of ASI, inspired the founding of AI companies, helped those companies get tens of millions in funding, and so on. To say that early TESCREALists were prescient would be like saying that I was prescient at 8am this morning when I prophesied that I would soon be drinking coffee — and then got up and made myself a cup of Joe.

***

Around 9:10, Tiku says that EA was initially focused on “global poverty.” I don’t think this is right. As Mollie Gleiberman argues, EA was longtermist — that is, focused on longtermists causes like AI safety — from the very start. To support her claim, she compiles a catalogue of copious examples from leading EAs around the time EA was founded, in which these EAs say that “existential risk” mitigation is what really matters. Global poverty was, basically, a means of marketing the movement. Once people had joined EA, they would then be exposed to arguments for why mitigating existential risks is actually the best way to “do the most good” in the world. Gleiberman describes this as a “Trojan horse,” where the ultimate goal was to pursue what she describes as “transhumanitarianism,” a portmanteau of “transhumanism” and “humanitarianism.”

***

Tiku argues around 9:22 that EA then merged with the Rationalists and transhumanists. But, again, I don’t think this is right. EA arose out of the inchoate Rationalist movement, and it was influenced by transhumanism from the very start. Rationalist blogs like Overcoming Bias and LessWrong were the Petri dishes in which EA first grew, and the cofounder of EA, Toby Ord, had been publicly defending transhumanism with the most influential ideologue of the 21st century, Nick Bostrom, since at least 2006. Ord was a member of Bostrom’s Future of Humanity Institute, founded in 2005 to advance the TESCREAL worldview, two years before EA was officially founded in 2009.

This is part of Gebru and my point re: the TESCREAL thesis. These ideologies are all tangled up together. They are outgrowths of each other. They are different colors, reflected through a prism, of the very same techno-utopian posthuman eschatology. EA emerged from the libertarian transhumanist movement that constitutes the nucleus of TESCREALism.

***

At 11:08, Lorenz says:

You also kind of alluded to something, which is that it doesn’t seem like they’re as anti-AI as you’d think. A lot of them, it seems, are interested in this idea of an aligned AI, or at least this idea that like it’s not even necessarily that AI is bad, maybe, but it’s just that we don’t have an AI that’s working in the interest of humanity. Is that right?

Tiku responds:

Yeah, that’s such a good way of putting it. And this is one of the concepts that is I think hardest for people outside to grasp, because if you think that a technology is going to destroy the planet — destroy every human on Earth, cause this kind of existential risk — you would think that you would be doing everything in your power to stop it. But what you quickly realize is that there [are] many groups of people that are also … enamored with the potential for utopia from this technology. “Aligned AI” is one term and that means kind of aligning AI with human values. Again, a very very abstract concept, but basically it’s like [the ASI does] what the developers want. You have a morality system like ethics and you approach problems in a way humans might even though you don’t have any real world experiences or emotions of your own. Sometimes they used to call it “Friendly AI.”

I’m really glad that Lorenz and Tiku highlight this. As mentioned before, TESCREAL doomers are not anti-AI. Even the most hardcore doomer wants ASI as soon as possible. Specifically, they want ASI once the alignment problem has been solved. If a doomer like Yudkowsky were to become convinced that the alignment problem is solved, he would suddenly switch from screaming, “Shut it all down, now!!” (where “it” refers to capabilities research), to screaming this: “Pedal to the metal! Build ASI asap!”

Don’t be fooled by the rhetoric these people use. They like to “sell” themselves as anti-AI, which gets normies who hate AI on board because those normies think their views align. They don’t. Yudkowsky would say that delaying the creation of an aligned ASI is as wrong, morally speaking, as accelerating the creation of misaligned ASI.

Nor are most TESCREAL doomers pro-human. They advertise themselves as such, but scratch the surface of their views and you find a deeply anti-human eschatology. I’ve written about this at length before (see below), and will soon publish, on this newsletter, chapter 4 of my forthcoming book because it offers a comprehensive account of the intimate links between the TESCREAL movement and pro-extinctionism. Stay tuned for that!

In my view, pro-extinctionism is the most important feature of the TESCREAL worldview that journalists should be highlighting. Quite literally everyone in the TESCREAL movement is pushing for a future in which our species is sidelined, marginalized, disempowered, and ultimately eliminated. This outcome is an inevitable consequence of the posthuman eschatology that they embrace and are actively trying to realize through the creation of superintelligent artificial deities. If this isn’t worth shouting about, then what is? Altman himself holds this view, as I show in a recent Truthdig article.

***

I don’t have many comments on the rest of the video. The discussion is very good, and Tiku foregrounds a number of important issues, such as the incestuous funding channels linking hundreds or thousands of TESCREAL organizations (many focused on AI safety) with super-wealthy TESCREAL donors (such as Jaan Tallinn, Dustin Moskovitz, and Vitalik Buterin — recipient of a Thiel Fellowship).

Some of these donors, including Tallinn and Moskovitz, helped launch the ASI race that they now worry could destroy humanity and, along with us, the “vast and glorious” posthuman future among the heavens. There is a great irony to this: no single group has done more to launch, sustain, and accelerate the ASI race than the TESCREAL doomers!

Consequently, these same people are now pouring money into social media campaigns aimed at halting the ASI race. The Future of Life Institute, for example — which was cofounded by Tallinn and received $10 million from Musk, and another $500 million from Buterin — has offered grants of between $100k and $500k for projects aiming “to educate and engage stakeholder groups, as well as the general public, in the movement for safe, secure and beneficial AI.” A central thrust of the interview with Tiku was to point out that you may very well have watched a video on YouTube warning about the existential risks of ASI and not realized that, in fact, the organization behind it is just two steps removed from the AI companies themselves, the intermediate link being the tech billionaires who funded both.

Vox Publishes an Article on Pro-Extinctionism

Vox — which, recall, has a deal with OpenAI — recently published an article titled “The People Who Actually Want AI to Replace Humanity.” It explores the ideology of “successionism” (i.e., replace humans with AI), and I have to say that the content is strikingly similar to my many articles and podcast interviews on pro-extinctionism. The author, Sigal Samuel, discusses transhumanism, Ray Kurzweil (the singularitarian), Effective Altruism, cosmism, longtermism, Beff Jezos and e/acc, as well as Daniel Faggella’s “worthy successor” eschatology — all topics I’ve covered at length in much the same way she does in her article. She also highlights the links between “successionism” and eugenics.

Sound familiar? Much of the article is little more than a summary of my work, though she doesn’t cite me or mention the TESCREAL acronym — even though that’s what she’s talking about. See for yourself.

It’s good that more people are talking about this issue, and Samuel appears to be generally critical of the TESCREAL ideologies. Aside from not citing me (which, you know, whatever), my biggest criticisms of her article pertain to what it leaves out. Although she mentions Toby Ord at one point, she doesn’t explicitly say that longtermism is a pro-extinctionist ideology, too — indeed, it’s nearly identical with Faggella’s “worthy successor” idea, as shown in my last newsletter article.

Doing so would have provided an opportunity to distinguish between two types of pro-extinctionism: first, the view that humanity should be replaced by autonomous AIs (like ChatGPT), and second, the view that humanity should be replaced by AIs that we become. Longtermists like Ord want to replace humanity with digital posthumans (which I satirically call “digital space brains,” after my friend Keira Havens suggested “space brains”), where these digital posthumans are radically transformed versions of us. In contrast, people like Verdon and Faggella appear open to autonomous, distinct, completely separate-from-us AIs usurping humanity.

When one carefully examines the space of possibilities here, it becomes immediately obvious that the entire TESCREAL worldview is pro-extinctionist, though certain versions of pro-extinctionism are more flagrantly horrific than others. All TESCREALists are rooting for Team Posthuman, meaning that, at the very least, they want to create a new posthuman species to run the show. This is dangerous because, I’ve argued before, if anyone becomes posthuman, everyone (else) dies out. If posthumanity arises, our days are numbered.

A few of my articles on pro-extinctionism:

1. “Should Humanity Go Extinct? Exploring the Arguments for Traditional and Silicon Valley Pro-Extinctionism.” The Journal of Value Inquiry.

2. “Digital Eugenics and the Extinction of Humanity.” Tech Policy Press.

3. “Meet the Radical Silicon Valley Pro-Extinctionists!” Realtime Techpocalypse Newsletter.

4. “TESCREAL" (see “Pro-Extinctionism”). Oxford Research Encyclopedia of Science, Technology, and Society.

5. “The Endgame of Edgelord Eschatology.” Truthdig.

6. “Under a Mask of AI Doomerism, the Familiar Face of Eugenics.” Truthdig.

7. “The Perplexing Rise of Antinatalism in Silicon Valley.” Realtime Techpocalypse Newsletter.

8. “Silicon Valley Pro-Extinctionism.” Realtime Techpocalypse Newsletter.

9. “The Growing Specter of Silicon Valley Pro-Extinctionism.” Realtime Techpocalypse Newsletter.

10. “On the Extinction of Humanity.” Synthese.

And basically any podcast/radio interview I’ve done in the past 2 years!

When I started this newsletter back in August 2025, I promised that we’d be covering a range of topics that (a) almost no one else is talking about, and (b) will become increasingly discussed in the coming years. For example, I have argued that we will see more and more debates spill into the public arena in which participants (e.g., Silicon Valley dwellers) are arguing not about whether pro-extinctionism is bad, but about which type of pro-extinctionism is best.

I have not been “ahead of the curve” on most things, but I think my (and Timnit’s) discussion of TESCREALism and my subsequent emphasis on Silicon Valley pro-extinctionism have been. The fact that Vox is now covering the very same ideas that I’ve been exploring for years is confirmation of this!

Next article (Tuesday): chapter 4 of my book, which is almost 20,000 words. Hopefully, it will provide some useful clarity on the different types of pro-extinctionism and the ways it’s entangled with the race to build god-like AI super-beings.

As always:

Thanks for reading and I’ll see you on the other side!