AI 2040
As you may know, some of the authors behind the widely-discussed “AI 2027” paper published a follow-up titled “AI 2040.” One of the authors, Daniel Kokotajlo, was a buddy of mine when I lived in North Carolina. He even read a draft of my 2017 book on existential risks and provided comments:
Whereas AI 2027 focused on how things could go terribly wrong with ASI (artificial superintelligence), this new document outlines a “positive vision for what should happen instead.” Both documents contain little more than wild speculations paired with unfounded assumptions and unrealistic predictions. In the Postscript, the authors write:
We think the world is asleep at the wheel. People say the words “AI will be transformative” without thinking concretely and seriously about the implications of broadly superhuman AI.
Yet I see no evidence that the authors have thought seriously about the implications of their best-case scenario, which they call “Plan A.” Plan A is little more than the TESCREAL vision of the future and, if realized, it would likely have devastating consequences for humanity — for reasons I outline in my “Pro-Human Manifesto,” and which I’ll briefly touch upon below.
To give you a sense of the normative futurology behind Plan A, here’s how they describe “Life after ASI.”
The resulting plans are too many to number. There are all sorts of schools of thought about what to do with your galaxy of resources,1 including:
Defer to the Future: Some people decide to give most of their resources to their children (or other people in the future), and let them choose.
Human Flourishing: Set up worlds of normal, free people living ordinary lives. This is a harder problem than it first seemed, because human civilization has already become profoundly abnormal: by this point, human activities like art or science has already become dominated by machines. Some handle this by accepting that their galaxy will be filled with post-scarcity humans, able to pursue whatever interests they find most meaningful. Others decide to roll back technology to leave space for human labor.
Digital Human Flourishing: If humans living happy, flourishing lives is good, then why not increase the number of people living these lives? Brain emulations can be run for vastly cheaper than real humans: a planet-size computer could plausibly simulate the equivalent of a million planets with happy civilizations.
Acausal Trade: Some argue that there likely are aliens in distant unreachable galaxies and that we could use various mechanisms to make deals with these aliens. If this is true, they argue, we could cut a deal to pursue compromise values rather than narrowly maximising our own values. Therefore, we might, on net, be able to do the most good by pursuing not just our own values, but instead a compromise of a vast number of other civilizations.
Humanity spreads across space in a dizzying variety of forms and ways of living — more diverse than anything Earth alone could have produced.
Take note that, for TESCREALists like the authors, “human” does not mean “our species” but “our species plus whatever posthuman creatures might succeed us, even if these beings are wholly artificial and wildly different.”
This is one of the linguistic tricks these people use to make their views sound more reasonable than they are. “We want to avoid human extinction” sounds good until you realize that their definition of “human” entails that our species could literally die out next year without “human extinction” having occurred. So long as our species is replaced by posthumans (that we either create or become), then “humanity” will persist.
The statements above are consistent with what Kokotajlo has said in interviews:
Here are a few reasons I strongly dislike the “AI 2040” analysis:
1. Extreme Hubris
They write: “First, our goal was to construct a positive vision, a grand plan for how to achieve an actually good future for everyone.” The extreme hubris of this statement is difficult to overstate. If I may use some vulgarities to get the point across: Who the fuck are they to be declaring that they know what “an actually good future for everyone” looks like?
(Sam Altman himself once said something similar when lying about how OpenAI will include people from around the world on its board. He said: “We’re planning a way to allow wide swaths of the world to elect representatives to a new governance board. Because if I weren’t in on this I’d be, like, Why do these fuckers get to decide what happens to me?”)
This is classic TESCREAL: the entire movement is infatuated with the supposed “intelligence,” “rationality,” and “brilliance” of its members. This enables them to believe that, because of their “intelligence,” “rationality,” and “brilliance,” they can speak for everyone in outlining a grand teleological vision of our collective cosmic future. These people think they have special access to fundamental truths about what the future ought to be, and by virtue of this prophetic knowledge their vision is superior to that of the benighted masses. It is a kind of messianic hubris, a radical form of elitism — nothing less.
In contrast, consider my favored approach to, as it were, designing a genuinely good future for everyone. I would argue that no single group should have the power to unilaterally impose their vision on everyone else. Rather, our collective future should, from the very start, be determined through a thoroughly democratic process. The people — you and I — should have a say in how things turn out. We should have a say in how advanced AI is developed, and whether it’s even developed in the first place.
As my friend Monika Bielskyte likes to say, “You cannot design for, you must always design with.” That succinctly captures the essence of my position, and there is not a single trace of this democratic approach in the “AI 2040” document.
Nowhere do they acknowledge that people outside the dumb TESCREAL movement should have a say in whether superintelligence is birthed in the bowels of an AI laboratory. Nowhere is there any indication that the authors are even curious about alternative visions of the future. The authors, I think, are completely unaware of just how much their preferred vision has been intimately shaped by the values of the milieu of Western capitalism plus utilitarian ethics (as I’ve written before, these are essentially the same). Maybe they don’t care.
To be clear, I am not saying that the authors shouldn’t express their opinion about which future is best. The whole point of establishing a governance mechanism to democratically determine the future is for people — or representatives — of every nation, tribe, religion, tradition, race, sex, gender, socioeconomic demographic, political party, ideology, and so on, to have a voice in the process of future-designing. That includes the voice of Kokotajlo and the other TESCREAL utopians.
But this is not the point of Kokotajlo’s vision. He and his coauthors are laying out a roadmap for the AI companies, policymakers, etc. to follow: slow down the ASI race, solve the “alignment problem,” and then create their version of a cosmic “utopia.”
Imagine, for a moment, that a group of religious extremists in Iran announce they’re building a new technology that could determine the entire future of humanity. There’s a good and bad way to build this technology, they claim, and Iranian scientists are working hard to ensure that it’s built the “right way.” These people then outline a roadmap for the future based on Islamic principles, which they declare will be “good for everyone.” The obvious response is to shout: “Who the hell are you to speak for everyone?” Yet, there is no structural difference between what the “AI 2040” authors are doing in their document and what these religious extremists are doing in this fictional example.
I genuinely can’t comprehend the level of hubris underlying “AI 2040.” Indeed, the hubristic heart of the TESCREAL movement is one reason I left circa 2019. These people have no idea how big the world is, yet feel wildly confident in their ability to design beneficial-for-all futures. Nonsense.
2. Climate Change
In their preferred future, Plan A, there is literally no consideration of climate change. Not once do they mention global biodiversity loss or the sixth major mass extinction of the last 3.8 billion years. How can you present yourself as a serious scholar of the future and not mention the environmental crisis?
I’m in the foothills of the Alps right now, and yesterday it was 97F (36C) — roughly 20 degrees over the average high for August. Half of the world is on fire. And 2027 is predicted to shatter temperature records around the world (many of which have already been broken this year) due to a historically unprecedented Super El Niño. By 2040, large parts of the world will be more or less uninhabitable — yet the authors say literally nothing about this.
Why? The main reason concerns “magical thinking” about ASI paired with an “engineering mindset.” These people simplistically see all problems as engineering problems — that’s why Nick Bostrom talks about a “solved world” after ASI arrives. If we have a superintelligence, then we automatically have a super-engineer; and if we have a super-engineer, it will be able to solve every engineering problem facing us — which is to say, once again, all problems. (Bostrom even talks about “paradise-engineering.”)
Climate change is no different. The only reason we haven’t solved climate change, on their account, is a lack of intelligence (and hence engineering). Once we have superintelligence, it will find some brilliant technical solution and immediately reverse all ecological pollution, restore ecosystems, remove all excess CO2 from the atmosphere, and so on.
This is magical thinking. The truth is that a lack of intelligence is not the reason climate change hasn’t been solved. Building a superintelligent machine isn’t going to somehow fix the problem before 2040. By then, Earth will be burning, millions or billions of people will be dying or migrating, and entire societies will be collapsing. It’s truly abysmal that the authors don’t take climate science seriously.
3. No One But the TESCREALists Want AI to Run the World
The authors write that in 2033,
as AI capabilities improve throughout Plan A, they increasingly shape every facet of life. … AI agents now form a population of two hundred million virtual workers that think and act 50x faster than humans, and never sleep. … By early 2036, there are 200 million AIs, equivalent to a workforce of around 100 billion humans, and 2 billion robots with some mixture of humanoid and other specialized form factors. … Gradually more institutions and equipment are handed over to AIs, such that it’s no longer true that humans could shut it all down if they wanted to. In fact it’s extremely untrue: Soon many of the world’s militaries are autonomous, run by AIs sworn to uphold various constitutions and treaties.
Why on Earth do the authors assume that people won’t fight back against the growing dominance of AI and attendant disempowerment of humanity? Of course there will be mass protests, civil unrest, and outrage. Depending on the details, there may be violent conflict in the streets as Butlerian jihadists launch a war against the ascendant AI overlords taking over the world.
This is a major weakness in the authors’ analysis, and you can count me in as one of the pro-human fighters who will do whatever’s necessary, within the bounds of my partly deontological ethics, to prevent the situation described above. Frankly, the authors’ vision of things going right sounds like a dystopian nightmare to me.
4. Colonizing Space Would Result in Constant, Catastrophic Wars
The authors blithely assume that space colonization won’t be a catastrophe. They completely ignore the academic literature on this topic, with renowned political scientists like Daniel Deudney arguing (in Dark Skies) that the almost certain outcome of colonizing our solar system will be constant, devastating conflicts. I’ve written about this topic numerous times, such as here, though I would recommend the closing chapter of Deudney’s excellent book for the original, robust analysis.
Again, this is an example of magical thinking — or what I call the utopian mindset. This mindset gives utopians a license to ignore the messy details of the world by saying, “I don’t know how exactly this will happen. But future technology — especially machine superintelligence — will have magical powers that will somehow enable us to figure it out. Hence, we needn’t worry our pretty little minds about it right now.”
This phenomenon is similar to the “suspension of disbelief” in fiction, whereby one ignores the plausibility of details in a story. No one watches Game of Thrones and stops to ask, “Wait a moment! Are the wings of those dragons really large enough to carry their massive bodies through the sky?” That’s an absurd question.
Similarly, those who embody the utopian mindset habitually dismiss questions about how exactly ASI will reverse climate change, upload our minds to computers, and colonize the universe without triggering catastrophic wars as absurd.
5. ASI Is Fundamentally Uncontrollable
The authors assume that there is a way to control ASI. We just haven’t discovered it yet. That’s why we should slow down ASI capabilities research: to give us time to solve the “alignment problem.”
But why think that ASI could ever be controllable? We’re talking about a perpetually evolving computer program with, by definition, far greater cognitive abilities than all of humanity combined. And yet we will somehow still control this infinitely self-improving beast?
Roman Yampolskiy (another guy I knew from the TESCREAL community!) is probably right that this is impossible, in the very same sense that a perpetual motion machine is impossible. My friend Remmelt Ellen has written about it here.
6. More Magical Thinking
The authors write that around 2036,
many of the world’s evils have dramatically reduced. Malnutrition, lack of medicine, and homelessness are nearly banished. Many diseases have been cured. Crime rates are lower than ever before. … People are wealthier and have more free time, plus there are new technologies that help, like AI matchmakers, better medicine, and AI tutors. And of course even the poorest people can now afford exotic vacations, amazing games, and enthralling entertainment.
Um, okay? Again, magical thinking enabled by the engineering and utopian mindsets. No serious discussion of how the tech elite will exploit ASI — if controllable — to permanently entrench their positions of power, control, and domination in society. The authors need to do a lot more to convince thoughtful readers that this is even remotely plausible.
7. There Is No “View from Nowhere”
The authors write about 2036, just 10 years from now!:
There are still problems in the world, and you can still contribute to solving them, partly by donating or volunteering but primarily by being politically active. Your vote is your most important asset.
The honest AI forecasters are helpful here. Years ago, if an AI said that one presidential candidate was better than another, people would suspect bias and the embarrassed company would retrain the AI to evade such questions. Now, thanks to transparency and improved alignment techniques, there are much smarter AIs that people can see don’t have any biases trained into them, that have built up an excellent track record over several years. When they weigh in on policy questions, people listen, especially when different AIs trained by different companies converge to the same answer.
AIs with no biases! Think for a moment about what that means. It assumes that there is a perfectly objective view from nowhere that, as such, is completely free of biases. The problem is that there is no view from nowhere. All views are views from somewhere. Not even science is perfectly objective.
Imagine asking the AI whether one should vote for a presidential candidate who’s a fascist and white supremacist. What would an unbiased answer to this look like? There is no possible response that isn’t biased in some way.
For example, a “no” response would be biased toward views that I hold, e.g., that fascism and racism are bad. I consider those biases epistemologically and ethically justified, which is why I hold them. But there’s no mind-independent fact of the matter that one can point to in showing that my biases against fascism and racism are correct. There are only more or less compelling arguments.
Later on, the authors write that
during the 2036 election season, voters are unprecedentedly well-informed, and the politicians that win have a genuine commitment to responsible stewardship of the upcoming singularity.
This assumes that all voters want to be well informed. Why think this?
Even more, studies show that climate deniers in the US are often just as knowledgeable about climate science as those who accept the climatological consensus (i.e. lack of education isn’t the best predictor of denialism — ideology is), meaning that being more informed doesn’t imply that voters will make wise or judicious decisions at the voting booth.
8. Woke AI?
The authors write:
Unless the alignment science is somehow all wrong, typical AIs are now more virtuous than the most virtuous humans. This alone has profound effects: it’s as if saints and angels were walking among us.
What on Earth do they mean by “virtuous”? Are they saying that future AIs will be “woke” — because that’s how I’d understand “virtuous.” This is just inane drivel.
9. Butlerian Jihadists, Unite!
They say that in 2039:
AIs have become increasingly load-bearing in all facets of society, from business to politics to even (some parts of) the armed forces.
Again, the vast majority of humanity will likely revolt against this. Most people aren’t going to want AI to take over “all facets of society.” More probable is that there will be blood (and microchips) in the streets as the Butlerian jihadists fight against the AI takeover.
Conclusion
I could go on, but it would be tedious. “AI 2040” is not a work of serious scholarship. It’s not written by genuinely thoughtful people who are aware of their own biases — of the extent to which their “utopian” vision for the future is rooted in white, Western, capitalist values. (Indeed, all the authors are white men. Any surprise there?)
The document is severely lacking in viewpoint diversity, and the hubris behind their prescriptions for the future is staggering. It’s a symptom of how impoverished the mainstream debates about AI and humanity’s future that documents like this get attention.
What have I missed? What do you disagree with here? How might I be wrong? Please share your thoughts below! As always:
Thanks for reading and I’ll see you on the other side!
Lots of “schools of thought” — this is a tiny list of schools of thought within the TESCREAL movement. As I point out below, the authors seem utterly oblivious to the vast diversity of opinions from outside the TESCREAL community about what the future should look like.
After reading Yudkowsky's book on AGI I think its obvious these guys belong in the writing room of a shitty Hollywood movie. Very irritating these guys have any influence at all!
Robin Hanson also had an article on government against the prospect of space travel on grounds that it'd lead to new factions outside of their control which might turn against them.
https://www.overcomingbias.com/p/most-likely-future-filter-world-govt