Political Optimism and the Underground Seattle Scene
We begin with this:
I am — we all are — so accustomed to the interminable flood of relentlessly awful bad news that I forgot what it’s like to feel, you know, hopeful and optimistic.
To be clear, the struggle is only just beginning: the Republicans and their allies in Silicon Valley have left a monumental mess to clean up. Incoming Democrats will need to address, with urgency, the rapidly worsening climate crisis, the out-of-control AI race, the war with Iran, and Trump’s unprecedented corruption — not to mention passing a medicare for all bill and restoring severely damaged democratic norms. Hakeem Jeffries and Chuck Schumer need to go, and the new cohort of energetic, charismatic young politicians need to take over. (We’ll see.)
The political reorientation of the US also opens up a universe of new possibilities for writers, activists, and others to shape the AI race. As mentioned in a previous article, I’m in the midst of founding an anti-TESCREAL institute, which is still nameless. Post-midterm elections, the feasibility of influencing young progressives entering the House and Senate has increased greatly. I think they’d be open to hearing what the institute has to say about the dangerous ideologies driving current efforts to build a Digital God. (Contact me if you’re interested in funding us! We should have a website up soon. :-)
Pair this with the fact that the Overton window continues to shift toward TESCREAL. I previously mentioned that the Economist article about EA (Effective Altruism) being the “biggest idea” of the century was in part about longtermism and transhumanism — the latter, of the Extropian variety. The journalists who wrote that article were so close to connecting the dots: EA in the context of Silicon Valley means “longtermism,” and longtermism is just libertarian (Extropian) transhumanism plus space expansionism, built on the “ethical” foundations of utilitarianism.
The Overton window has already shifted to the point that, as someone alerted me just yesterday, zines are popping up in Seattle explicitly discussing the TESCREAL worldview (from here and here):
NYT: Dario Amodei Is a TESCREAList
Another example of the Overton window shifting comes from a just-published New York Times profile of Dario Amodei. In it, the authors talk about his involvement in EA and Rationalism, how the work of Ray Kurzweil and Eliezer Yudkowsky crucially shaped his worldview, and his long-time worry that the Singularity could go awry.
Here are the highlights, interspersed with some commentary:
One night in 2007, Dario Amodei regaled a friend who was visiting him at Princeton University with his thoughts on “The Singularity Is Near,” a 2005 treatise by the computer scientist Ray Kurzweil.
This is also the book that got me interested in transhumanism circa 2006. Initially, I was mortified by Kurzweil’s singularitarian vision of the future, and especially his claim — from his previous book, The Age of Spiritual Machines, which I subsequently read — that “a planet approaching its pivotal century of computational growth — as the Earth is today — has a better than even chance of making it through. But then I have always been accused of being an optimist.” Up to a 50% chance of not making it through due to advanced technology sounded outrageous to me: maybe we shouldn’t develop these technologies!
However, over time, The Singularity Is Near convinced me that the enterprise of science and technology is a juggernaut that, as such, nothing can stop. ASI (artificial superintelligence), posthumanity, etc. are “inevitable,” as Kurzweil puts it. That realization — which I now reject — led me to embrace the Singularity, and indeed 10 years later I became a research assistant for Kurzweil’s follow-up, The Singularity Is Nearer.
Continuing with the article:
Dr. Amodei had just read the book, in which Mr. Kurzweil predicted that artificial intelligence would eventually surpass the human brain in a moment known as the Singularity.
That’s not how Kurzweil defines the Singularity. He writes that the Singularity is
a future period during which the pace of technological change will be so rapid, its impact so deep, that human life will be irreversibly transformed. … The key idea underlying the impending Singularity is that the pace of change of our human-created technology is accelerating and its powers are expanding at an exponential pace. Exponential growth is deceptive. It starts out almost imperceptibly and then explodes with unexpected fury — unexpected, that is, if one does not take care to follow its trajectory.
Back to the NYT profile:
As he twisted his hair with his fingers, Dr. Amodei told his friend, Colin Heckman, that the milestone could happen within their lifetimes.
“He became obsessed,” said Mr. Heckman, who went to high school with Dr. Amodei in San Francisco. “He said A.I. had to be built right, or we’re going to be done.”
Dr. Amodei, then 24 and a doctoral student in physics, was in a period of soaking up information. He had become absorbed in the obscure blogs “Accelerating Future” and “Overcoming Bias,” where writers discussed A.I.’s potential to wipe out humanity. And he had delved into the online forums of GiveWell, an organization that aimed to maximize the effectiveness of charitable giving, where intellectuals debated how to make the most money for philanthropy.
Note that Overcoming Bias is where Eliezer Yudkowsky blogged alongside “America’s creepiest economist,” Robin Hanson, until he founded LessWrong circa 2009. Hanson is a TESCREAL pro-extinctionist and Men’s Rights activist who proposed the idea of “sex redistribution” in the aftermath of an incel terrorist attack, published a blog post in 2010 titled “Gentle Silent Rape,” and once claimed during a debate that “‘the main problem’ with the Holocaust was that there weren’t enough Nazis!” But I digress!
These books, blogs and online communities have influenced Dr. Amodei to this day, according to interviews with more than two dozen of his classmates, friends, teachers and colleagues. They help explain the ideas that drive the 43-year-old chief executive of Anthropic, a company that is set to conduct one of the world’s largest public listings this month, and offer an understanding of how the man at the forefront of the A.I. revolution actually thinks about A.I.
Take note of the words, “to this day.” The article continues:
At the heart of Dr. Amodei’s worldview, his friends and colleagues said, are two interwoven philosophies that share an uncompromising belief that mathematical reasoning can change the future of humanity.
One is “rationalism,” which aims to apply dispassionate thinking to the world. Espoused by people like Eliezer Yudkowsky, a self-taught A.I. researcher, it argues that A.I. will become powerful enough to destroy society. The other is “effective altruism,” which evolved alongside rationalism and seeks to maximize social good with mathematical precision.
Indeed, EA — which was longtermist from the very start1 — was incubated on Rationalist websites like Overcoming Bias and LessWrong.
These schools of thought fused for Dr. Amodei into an unyielding conviction that A.I. could be so dangerous that only he and like-minded technologists could be entrusted with building it.
That catalyzed his stated efforts to create A.I. before people without concerns about the safety of humanity did, and spurred a desire to control the technology, his friends and colleagues said.
Extraordinary elitism and hubris!
With A.I., “the benefits will only be achieved if we build the technology in the right way, and — so long as we use the time we gain well — it is worth taking unusually deliberate care to get it right,” Dr. Amodei wrote in a September essay that called for a slowdown in the technology’s development.
Take a step back: Rationalism directly emerged out of the transhumanist, Extropian, and singularitarian movements, which were nearly coextensive. (Extropianism is a form of libertarian transhumanism. An Extropian coined the term “singularitarian.” All singularitarians are transhumanists. Etc.)
Rationalism emerged from Yudkowsky’s writings around the time he created LessWrong, and Yudkowsky himself is a transhumanist who participated heavily in the Extropian movement and was once the leading singularitarian next to Kurzweil. Hence, when the NYT talks about “Rationalism,” it’s really talking about “Rationalism, transhumanism, Extropianism, and singularitarianism.”
The same goes for EA. To reiterate what I said at the beginning of this article, in the context of Silicon Valley and the ASI race, “EA” almost always means “longtermism,” and longtermism is just Extropian-transhumanism plus space expansionism (a central theme of singularitarianism), built on an “ethical” foundation of utilitarianism. More concretely, longtermism was cofounded by Yudkowsky’s collaborator, Nick Bostrom, who also participated in the Extropian movement and has been writing about the Singularity since the 1990s.
We should also note that Amodei isn’t the only AI company CEO that the NYT has described as having emerged from the nearly identical EA and Rationalist communities. Here’s what it said in a 2023 profile of Sam Altman:
[He’s] the product of a strange, sprawling online community that began to worry, around the same time Mr. Altman came to the Valley, that artificial intelligence would one day destroy the world. Called rationalists or effective altruists, members of this movement were instrumental in the creation of OpenAI.
Back to the article on Amodei:
These ideas formed the foundation for Anthropic, which Dr. Amodei established in 2021 with his sister, Daniela Amodei, and five others who spent time around rationalist and effective altruist communities. And as the company sprouted, their views — which Dr. Amodei has pushed vocally and relentlessly — have become increasingly prominent. …
At Lowell High School, San Francisco’s top public high school, Dr. Amodei was a voracious learner who took classes at the University of California, Berkeley. He pondered ideas like utilitarianism, which argues that people should maximize happiness for the most living beings, and worried about issues like the Y2K bug and its potential to ruin the world’s computers, classmates said. …
On one trip, he went with Mr. Avery and Mr. Heckman to a sushi restaurant, where they debated utilitarianism. Dr. Amodei was a vegetarian but argued it was fine to eat crabs because they are invertebrates.
During the meal, Mr. Heckman presented a thought experiment: If someone started killing as many chickens as possible, when would it make sense to kill that person to prevent more chickens from dying? Dr. Amodei chuckled, then said that “there was some number of chickens that would outweigh a human in the utilitarian sense,” Mr. Avery recalled.
Utilitarianism is just ethics as a branch of economics. Value (welfare or wellbeing, according to utilitarians) is a quantifiable entity that must be maximized, and determining the morally “right” action involves nothing more than entering numbers into an expected value equation. (You can see why this appeals to people in STEM, despite being widely rejected by moral philosophers.) It is a highly simplistic, profoundly dangerous approach to thinking about ethics.
Not to digress too far, but here’s an illustration: utilitarianism provides an “ethical” reason to pursue ASI even if there’s a 10-25% chance of total annihilation (that is, final human extinction), as Amodei estimates. Why? Because the value of the future, once we become digital posthumans and build a sprawling multigalactic civilization full of trillions of people living in computer simulations, is claimed to be absolutely enormous. Hence, if you plug the numbers into your expected value calculator, you find that the expected value of moving forward is far, far greater than the expected value of never building ASI (or even delaying ASI).
This logic holds even if one believes there’s close to a 100% chance of total annihilation in the near future. As someone recently posted on X:
Spoke to an Anthropic employee today. They have a near-100% p(doom), but they think the only chance of survival is that they’re somehow completely wrong about alignment and it turns out to be solvable, and the only way to get there is by accelerating capabilities as fast as possible. They were much more confident that humanity would never have the political will to do what needs to be done to stop ASI development than that alignment would be solved, even though they think alignment will almost certainly not be solved.
Bostrom himself recently argued that we should push ahead even if there’s up to a 97% chance of ASI annihilation — again, because the expected value of a “good” future completely overwhelms the badness of everyone dying near-term. This is pernicious utilitarian thinking, also on display when Evan Hubinger, Anthropic’s Alignment Science team, affirmed last September that he personally thinks there’s at least a 10% chance of universal death in the coming years.
That’s one way these people are able to wake up in the morning and think, “I might be contributing to the annihilation of humanity, but this is compatible with me being a morally good person — because my work on AI has the highest expected value of all options available.”
Returning to the NYT article:
By then, Dr. Amodei was embracing the writings of Mr. Kurzweil and Mr. Yudkowsky, a leader of the rationalists. He was especially intrigued by Mr. Kurzweil’s idea that advanced technologies would let humans exist beyond natural limits, a belief known as transhumanism.
Like Yudkowsky and Bostrom, Kurzweil participated in the Extropian movement, and hence the variety of transhumanism being referenced here is Extropianism (and singularitarianism). The NYT article has now referenced, explicitly or implicitly, the TESREAL ideologies — all that’s missing is the “C” for “cosmism,” though cosmism is basically just longtermism without the utilitarian base (though it’s compatible with utilitarianism).
During Mr. Heckman’s 2007 visit to Princeton, Dr. Amodei told him that the Singularity would happen by 2040 and that humans might someday meld their minds with machines. “We would over time become the new superintelligent A.I. or A.I.-human hybrid,” Dr. Amodei said, according to Mr. Heckman.
This gestures at what I’ve called Silicon Valley pro-extinctionism, which is intimately bound up with the TESCREAL worldview: after the Singularity, the human species will be replaced by posthumans in the form of uploaded minds and/or cyborgs. Altman has said the same thing: to survive the advent of ASI, we’ll need to “merge” with machines. Biological humans will disappear through replacement with some form of digital posthumanity. Same posthuman eschatology.
It’s clear that Amodei still embraces these transhumanist ideas, by the way, as he’s written in recent blog posts that ASI may enable us to “transcend” our biology through mind-uploading. That’s Kurzweil’s promise — it’s what he claims will happen when the Singularity occurs in 2045.
Dr. Amodei became so fascinated by this thinking that he encouraged his friends and sister to attend the annual conference of Mr. Yudkowsky’s Singularity Institute. (The group is now called the Machine Intelligence Research Institute.)
This must be a reference to the Singularity Summit, cofounded by Yudkowsky, Kurzweil, and Peter Thiel.
Mr. Yudkowsky was “one of the deepest thinkers I have ever encountered” and “explicitly trying to bring about a singularity that will be beneficial to humanity,” Dr. Amodei told friends in a 2008 email seen by The Times.
I have to say: even when I was a hardcore TESCREAList, I never understood the appeal of Yudkowsky. I couldn’t make sense of why so many people lionized him and worshipped his writing. To this day, he’s one of the only people whose blog posts I read and repeatedly think, “I have no idea what he’s trying to say here.”
In 2010, he wrote that he was “considering a career in finance, which would allow me to give away more money than I currently do,” according to an email seen by The Times.
Hence, Amodei — who was among the very first EAs — considered pursuing the “earn to give” career path. This is what Sam Bankman-Fried did: he worked at Jane Street Capital, then took a job at the Centre for Effective Altruism, after which he went into crypto. Many EAs have and still do work at Jane Street. I suspect Amodei was considering a job there, too.
In a blog post for GiveWell that year, Dr. Amodei discussed a decision to give away $10,000. He was choosing to donate to either the Stop TB Partnership, which focused on preventing adult tuberculosis, or VillageReach, which helped vaccinate children in African villages.
The Stop TB Partnership, Dr. Amodei wrote, was superior in “cost-effectiveness” because it served adults and “adults are capable of deeper and more meaningful experiences” than infants. “I think an adult death is perhaps 2 or 3 times worse than an infant’s death,” he added.
This is yet another illustration of the poverty of utilitarianism: value is assumed to be quantifiable. At first glance, the way utilitarians mathematize morality can give their view the appearance of rigor. Yet the numbers they use are all, more or less, completely made up — drawn from a hat. One could just as easily argue that infant deaths are “2 or 3 times worse” because infants have more life ahead of them, and hence could potentially realize more utils (units of utility, or value).
This gestures at a broader important point about utilitarianism: one can get the numbers to support just about anything. From a utilitarian-longtermist perspective, an all-out nuclear war that kills 7 billion people might be good because it could convince those who survive to take “existential risk” much more seriously, thus increasing the probability of trillions and trillions of digital posthumans in the far future. You can get that argument to work, and hence longtermists should advocate for nuclear war!
In practice, utilitarianism is useless — often utilized by people to provide “ethical” justification of positions they pre-theoretically prefer for non-moral reasons. “I want to believe this, so I’ll make up some numbers to justify that belief.”
Back to the profile:
In 2011, he started postdoctoral work at Stanford studying cancer cells. In his personal life, he experimented with anti-aging supplements, maintained a strict swimming regimen and became interested in caloric reduction, which was shown in some studies to prolong the lives of lab mice, his friends said.
These are common interests among transhumanists. I myself considered taking anti-aging supplements years ago2 (Kurzweil used to boast of consuming 250 supplements every day), as well as caloric reduction to extend my lifespan. I strongly suspect that Amodei has also at some point tried nicotine chewing gum, which Nick Bostrom claims helped him write his 2014 book Superintelligence, and modafinil, a “nootropic” that was all the rage among my TESCREAL colleagues in the 2010s.
The article then mentions how Dario Amodei was roommates with Holden Karnofsky, one of the most influential architects of the EA-longtermist ideology who now works at Anthropic:
At the time, Mr. Karnofsky was dating Ms. [Daniela] Amodei. The couple moved into a group house in San Francisco, which eventually included Dr. Amodei; Jared Kaplan, a theoretical physicist; and Christopher Olah, a college dropout who had been tutored in math by Dr. Amodei.
The housemates discussed Mr. Yudkowsky’s essays and explored rationalism and effective altruism, three people who visited the home said. They also attended Burning Man, the annual counterculture gathering in the Nevada desert, sometimes dressing up in animal costumes.
So, they lived in an EA-Rationalist group home, which are common in the Bay Area.
Conclusion
If there was any doubt that Amodei is a TESCREAList, this article should put it to rest.
It also highlights the fact that — as I’d put it — no single individual has done more to launch, sustain, and accelerate the ASI race than the doomer-in-chief himself, Yudkowsky.
He inspired Amodei, Altman, Musk, Shane Legg, and Demis Hassabis to pursue ASI. He introduced Hassabis to Thiel, thus enabling DeepMind to get funded. He brought the idea of ASI to people’s attention like nobody else via his prolific, rambling blog posts, Harry Potter fan fiction, and the Singularity Summit. He encouraged a whole generation of young people to pursue jobs at the AI companies — some of the very people who now claim that the technology they’re actively trying to build will kill every person on Earth by the early 2030s.
Yudkowsky has tried to deny his role in igniting the ASI race, but the facts are clear: he is, to a large degree, responsible for the current mess we’re in. (There’s a lesson here: don’t trust his judgment moving forward.)
As it happens, a video recently resurfaced in which Amodei says that ASI could very well annihilate the entire human population:
In my current tentative opinion — bringing this full circle — one of the first things that Democrats need to do once they resume power is to impose a complete and permanent ban on all projects aimed at triggering RSI (recursive self-improvement) and creating ASI. Those who violate this rule should be imprisoned for decades.
Democrats should then strive to establish a global treaty that prohibits anyone, anywhere from building ASI. To be clear, one doesn’t need a precise definition of “ASI” to back this plan. The point is that nothing good will come of someone building a system that can outmaneuver us in every important respect, solve complex problems much faster than us, set and modify its own goals without human intervention, act as an autonomous agent in the world, etc.
If such a system were possible to build and were actually built, I don’t see how the outcome would be good for us.
But what do you think? As always:
Thanks for reading and I’ll see you on the other side!
As Mollie Gleiberman writes:
From the beginning, EA discourse operated on two levels, one for the general public and new recruits (focused on global poverty) and one for the core EA community (focused on the transhumanist agenda articulated by Nick Bostrom, Eliezer Yudkowsky, and others, centered on AI-safety/x-risk).
I didn’t because I worried about contamination.
Quite frankly (as you know) I personally do not spend much time thinking about the dangers of ASI (as-a-real-entity) insofar I don't believe that we are anywhere NEAR such a capability and take any news of speeding up towards RSI with not only a grain of salt but rather a truckload thereof.
HOWEVER, if I was willing to grant that there was actually such a thing as a hyper-intelligent selfimproving system that rapidly tumbles towards a kind of omniscience (And I am not. For a number of reasons, only some of which have to do with the current techstack and some of which have to do with what I believe is Günther's adequate assessment of ontology in the age of cybernetics, but I digress)...
in any case, IF I were to believe that, then I would still not be particularly concerned. This has something to do with the fact that I not-so-secretly have the belief that knowledge is not as neutral as the instrumentalist view makes it seem. We are raised in a society that allows us (at least in principle) to see people that are chasing after absolutely self-defeating goals as smart or worse, business savvy. I think that says more about the absolute rottenness of our understanding of epistemology than about the discipline as such.
I think there is something to be said toward the old forms of wisdom and their necessary connection to knowledge... all of this just to say, I believe if you were really in a situation in which you would create such an apparatus, they would be far beyond interested in harming us and yet not so far beyond that they don't think about it.
This is neither a lovecraftian elder god nor a Terminator. This would be a machine that actually tries to work with the world. And I think the very structure of knowledge is biased towards making the world whole. "Totum relucet in omnibus", the Cusanus word, would need to apply for this machine after all anyhow: everything would need to be understood through everything, which would also include the different points of view, which would entail a sort of bias towards protection.
This is an almost theological position, sure.
But if we are going to go into the realm of wildspec you might as well go there.
Because the world is actually a secretly very friendly beautiful place with such nice surprises. Sometimes.