tl;dr
There is no contradiction between the following statements:
AI models are becoming increasingly powerful.
These increasingly powerful AI models pose significant risks to society, perhaps even humanity itself.
None of this is bringing us any closer to AGI, ASI, or the Singularity.
In other words, one can accept that AI systems are advancing quickly, these advances pose serious risks, and ASI remains as distant today as it was, say, in 2022. This article explains why.
“Much Less Weird than it Seems Like It Should Be”
As we’ve discussed on numerous occasions, many people in Silicon Valley are convinced that the Singularity is imminent. Sam Altman wrote last year that “we are past the event horizon; the takeoff has started.” In January, Musk posted on X that “we have entered the Singularity.” He reiterated this in early August, after news about OpenAI’s Astra model (discussed below) broke, saying:
(Since he asked, the temperature over here in Europe is UNBEARABLE. Good thing that AI isn’t contributing to climate change!)
Demis Hassabis, who recently stepped down as CEO of DeepMind, said that “ten years from now, I think we’ll realize that we were standing in the foothills of the singularity now.” He believes ASI will arrive by the early 2030s, after which we will quickly start building a Dyson swarm around the sun (lol). And Dario Amodei claims that AGI could arrive later this year. Even Jensen Huang of NVIDIA said last March: “I think we’ve achieved AGI.”
Amusingly, after Altman proclaimed that we’re “past the event horizon” in his blog post, he added that “humanity is close to building digital superintelligence, and at least so far it’s much less weird than it seems like it should be.”
Yeah, dude, it’s not that weird because the Singularity isn’t happening! The event horizon still looms above the actual horizon before us — which is to say, we’re nowhere close to triggering an intelligence explosion.
The OpenAI-Hugging Face Incident
Yet, it’s clear that AI models are becoming significantly more “powerful,” and that newer models — Mythos being the most obvious example — are introducing novel risks that should worry us greatly.
I’m sure you’ve heard of this incident — it’s one of several that have recently been reported by AI companies. Basically, OpenAI was testing “an experimental in-house AI, working with the latest commercially available model.” As part of the test, they gave it “a set of cybersecurity tasks” to complete. The AI then figured out a way to escape its digital sandbox by identifying and exploiting “a previously unknown zero-day vulnerability,” hack into Hugging Face, and search Hugging Face’s datasets for a solution to the cybersecurity problems it was tasked with solving.
This happened in part because OpenAI failed to design an adequate sandbox. Some of the AI’s security guardrails were turned off — if they had been on, they “may have prevented this” from happening — and the sandbox “had one small connection to the outside world, to allow the AI to download software it needed.”
This portal to the public Internet was mediated by a “package registry cache proxy,” which is basically an intermediary server that provides a kind of barrier between the AI and public Internet. But the AI found a vulnerability in this intermediary server and, in doing so, broke free of its prison, thus enabling it to access the Internet and hack into Hugging Face.
It’s worth highlighting that no one at OpenAI told the AI to cleverly break out of the sandbox and hack into another company — which constitutes a felony! Yet, as Gary Marcus notes:
the current incident was NOT an attempt where [the] system built a goal for itself or developed a motive; the system was following instructions, but not setting high level goals. It was not trying to take control of the world a la Terminator, it was just trying to cheat on a test, which is at least a bit less scary.
“On the less comforting side,” he adds that OpenAI’s guardrails (when turned on) are nonetheless
likely to be permeable, just like all guardrails anybody has built to date. Even putting aside the thorny questions of open weight systems, we have no guarantee whatsoever that future models won’t be able to do similar things, such as finding zero-day exploits to hack systems. To the contrary, we can expect more incidents of this type.
This is unsettling. As a New Yorker article puts it, what might have happened “if the rogue AI had had access to a robot, or a self-driving car, or a bio lab, or a weapons system”? Recall that Pete Hegseth is actively transforming the US military into “an ‘AI-first’ war fighting force across all components, from front to back,” according to an official document from the Department of War. Already, war game tests show that AIs will threaten nuclear war in 95% of scenarios.
Even more, Chinese companies are releasing open-source AI models that, as such, allow people to remove any guardrails they might come with. For all of these reasons, “we can expect more incidents of this type,” though we should also expect future cyberattacks to become more sophisticated and difficult to prevent.
AI Invents New Viruses
Around the same time this snafu became public, the New York Times reported that
Scientists at Stanford University and the Arc Institute, a research organization in Palo Alto, Calif., taught AI [called “Evo,” similar to ChatGPT] to recognize patterns of DNA structure in nature, and then to use that data to write recipes for entirely new viruses.
The researchers followed those recipes to create DNA molecules, which they inserted into bacteria. The modified bacteria then produced viruses never seen in nature. The viruses were able to infect other bacteria, demonstrating that they were viable. …
As Mr. King and his colleagues tested more genomes, they saw more clear dots. All told, they discovered that 16 of Evo’s genomes produced viable new viruses.
They proved to be as resilient as natural ones. In fact, some multiplied faster than Phi X-174 [see below]. “They’re not just sickly versions of stuff that already exists,” said Oliver Crook, a protein chemist at the University of Oxford who was not involved in the new study.
The article adds that, fortunately, “the viruses dreamed up by A.I. do not pose a threat to humans, because they are all similar to a naturally occurring virus called Phi X-174, which can infect only bacteria,” and they’re “not radically new creations. They tend to be very similar to natural species, relying on the same underlying biology.”
Nonetheless,
Evo’s initial success has raised concerns that A.I. could be used to create deadly pathogens. “You could say, ‘Hey, genomic language model, make me an influenza genome that is modified to be more transmissible or to be more lethal,” Dr. Hanke speculated.
The potential dangers were already on the minds of the researchers as they trained Evo. They did not provide the model with data about viruses that infect humans, and they excluded similar viruses that infect other animals, plants and fungi.
The possibility of AI-enabled designer pathogens has been discussed for decades among scholars of global catastrophic risks. An express goal of synthetic biology is to “black box” aspects of genetic manipulation: rather than spending years in a laboratory acquiring “tacit knowledge” about how to manipulate the building blocks of life, it aims to enable individuals to achieve the same ends by simply pushing a button. (This is also called “deskilling.”)
Furthermore, the genomes of horrific pathogens like Ebola and smallpox are publicly available online (I won’t include a link!), and there’s no theoretical barrier to combining the incurability of Ebola with the contagiousness of the common cold, lethality of Rabies, and long incubation period of HIV. Doing so would enable “risky agents” (a technical term meaning those with an explicit desire to destroy civilization and/or humanity) to synthesize a germ that spreads around the world undetected, infects nearly everyone, and then 10 years later results in mass death, perhaps even human extinction.
Is this hyperbolic? Maybe, a little — but the point is that AI could supercharge the risk. If risk = consequences x probability, then even a low-probability pandemic that kills a large fraction of the human population would still constitute a huge risk.
Add to this the fact that there is a very large number of people — a small percentage of the total population, but a sizable demographic in absolute numbers — who harbor a death wish for humanity. I’ve written about some of these people in the past, such as the Efilists, who say they’d murder everyone on Earth, including themselves, to eliminate all suffering. (This arises from their allegiance to negative utilitarianism.) An Efilist even bombed a fertility clinic in California last year.
The point is that initiating an engineered pandemic is becoming increasingly feasible. Deskilling, publicly available genetic data, tools like CRISPR-cas9, and now AI are a recipe for disaster.
AI and Mirror Life
What worries me even more than designer pathogens is the possibility of AI assisting in the creation of “mirror life.” This refers to forms of life that consist of molecules with the opposite chirality of the molecules that comprise all biological life on Earth right now. A bacteria with mirror molecules could potentially infect the human body without being detected by our immune systems — such bacteria would, essentially, be invisible to our internal defenses.
As an article in Science, written by nearly 40 prominent scientists, declared:
We cannot rule out a scenario in which a mirror bacterium acts as an invasive species across many ecosystems, causing pervasive lethal infections in a substantial fraction of plant and animal species, including humans.
How might AI exacerbate this risk? Amodei himself highlights the danger in his essay, “The Adolescence of Technology,” writing that a “sufficiently powerful AI model (to be clear, far more capable than any we have today) might be able to discover how to create it much more rapidly — and actually help someone do so,” where “it” refers to “mirror life.” Yikes.
AI Capabilities are Getting Spikier, Not More General
All of this leads to the linchpin of my argument here: advances in AI and the potentially catastrophic risks they’re introducing don’t for a moment imply that we’re getting closer to AGI, ASI, or the Singularity.
This is counterintuitive, but consider a recent X post from a professor at Cambridge named Adam Hunt. He reports having previously been bullish about AI, but now considers himself bearish. The reason is that
the whole “it turns out if you keep training and scaling the models more they develop broad new capabilities in lots of domains” thesis is wrong (sorry Demis). The recent batch of models haven’t got more general, they’ve got less general. This is most obvious in the fact that their language outputs have got much worse in comparison to e.g. o3. If they were gaining generalist capacities we would expect them to be describing their work in ever more graceful and comprehensive prose!
Take a look at this widely shared image:
The “mainstream AGI thesis” on the top hasn’t panned out so far. On this account, AI will initially show superhuman capabilities in narrow domains, but as research advances, its capabilities in other areas will also increase. Eventually, the AI will exceed humans in all cognitive domains of interest, at which point we’ll have ASI.
In fact, what appears to be happening is that AI is getting (significantly) better in narrow domains (e.g., coding, finding software vulnerabilities, inventing new viruses, etc.) without getting much better in all other domains — i.e., those it currently underperforms in. The spikiness of its capabilities is getting spikier. If the trend continues, it will become ever-more superhuman in particular domains without attaining general intellectual capabilities.
When someone asked about whether “chain-of-thought make[s] AI more general, less general, or the same?,” Hunt responded:
I dont know if ‘less general’ or ‘more general’ is the dimension that chain of thought works upon.
I think chain of thought basically exaggerates existing potency in the underlying models, by allowing that existing potential to reflect and re-reflect with its existing capabilities. So with coding that leads to much much better coding, because the models are good at coding.
But in scientific logic, which LLMs are not good at (e.g. in my work, recently Opus 5 tripped up over how seriously should it take one specific study of a particular drug’s effects on ADHD) then chain of thought doesn’t help - it gets trapped in illogical loops that it self justifies and can’t leave alone.
So no, I don’t think that scaling up inference compute will suddenly get you to AGI, because of fundamental spikes and flaws in the underlying training data/RL which lead to blind spots or weaknesses in the basic model, which can just compound over time in chain of thought rather than getting suddenly fixed.
This seems right to me. One can thus acknowledge major advances in AI capabilities while simultaneously holding that these advances aren’t inching us closer to ASI or the Singularity.
The Fallacy of Composition
Marcus makes a related point in a recent blog post, highlighting how AI hypesters are committing the fallacy of composition. For example, OpenAI reported that its new (unreleased) AI model called Astra “solved 10 major open problems in mathematics, quantum complexity, and theoretical computer science.” That led Dean Ball — among many others — to declare that:
The fallacy of composition occurs when “one infers that something is true of the whole from the fact that it is true of some part of the whole.” In the case of AI, the fallacious reasoning goes: “AIs are becoming more generally intelligent — thus getting us closer to ASI — because they are becoming more capable in this or that particular, narrow domain of cognition.”
That absolutely doesn’t follow. It’s like saying, “A particular aspect of my life is improving, as I now have more free time during the day to pursue hobbies. Therefore, my entire life is improving” — which of course might be false if one also just lost their job, spouse, etc. The increasing spikiness of AI capabilities doesn’t imply that more recent AI models are closer to ASI — superhuman general intelligence — than previous ones.
Conclusion
What’s the takeaway here? It’s that the incidents mentioned above don’t make me — and shouldn’t make you — more concerned that Altman, Musk, and the others are right in proclaiming that the Singularity is imminent or has already started.
At the same time, they should make us more anxious about the potentially catastrophic consequences of increasingly powerful spiky AI systems. An AI-enabled pandemic could be devastating. AI-enabled mirror life could literally destroy the biosphere. Cyberattacks will become more difficult to anticipate and prevent. And the fact that the US military is actively integrating AI into its infrastructure is hugely worrisome (as you may recall, Anthropic’s Claude was used in the illegal kidnapping of Nicolás Maduro and has helped select targets in Iran). Etc. etc.
So, I find myself both deeply worried about these new developments and no more nervous than I was about ASI suddenly emerging from the bowels of an AI laboratory. As I like to say:
We don’t need ASI for AI to destroy the world.
But what do you think? What have I missed? How might I be wrong? As always:
Thanks for reading and I’ll see you on the other side!
Something I wanted to point out with the AI Model at Stanford creating new viruses was the fact that when Stanford did this the AI Model created “thousands” of viruses (Or their sequences technically) of which the scientists only took 300 of these sequences. And out of those 300 only 16 were viable…that is only 5.3%. Now maybe if the sample group was larger the numbers would be bigger, and I’m also not saying this makes it better. But I think it’s important to note that not every sequence the model makes is viable. Which means that while the sequence looks legit, doesn’t mean it’s gonna work.
As for the “Mirror Life” section, while I’m not a Molecular Biologist, I have watched SciShow’s video on Mirror Life and at the top of that video is a very long comment from an actual Molecular Biologist who’s attempts to go into the reason why Mirror Life End-times scenario won’t pan out.
I do think it’s important to recognize our limitations in things. Émile, I love you 🫶❤️…but you’re not a Molecular Biologist (I mean that with respect btw!). And neither am I! I think it’s fair to be afraid of things, such as Mirror Life, but you and many others also taught me that AI is not a magician. Just because AI “Could” create “Mirror Life” doesn’t mean it “can”.
Great article as always though 🙂
Is language what makes humans smart?
No. Language is a large part of what makes us successful or dominant as a species but language is not what makes us smart.
The human brain is the product of >500 million years of selection for the physical and ecological environment and >50 million years of selection for the social environment but only ~100 thousand years of selection for the symbolic environment. Most of our intelligence and reasoning is pre-language and sub-symbolic. Language is mostly just a medium that we express our intelligence through or use our intelligence on. What language does most fundamentally is allow us the capacity to learn from the experience of others. In contrast, other animals may be quite intelligent but are limited to learning from their direct experience (which includes observations of and interactions with others but not descriptions of the experience of others). This is part of what makes humans uniquely unique compared to other animals. The capacity for language may require some threshold of intelligence and the capacity for language might increase the returns to intelligence in a positive evolutionary feedback but language is not the basis of our intelligence per se.
Is language evidence of intelligence?
Yes and no.
Language is evidence of intelligence in humans only when considered in context of and despite our particular limitations and constraints. Humans learn to use language with limited and imperfect exposure and a limited amount of resources. Once we've learned the basic structure of language we can learn new words and concepts from a very limited number of exposures (and can continuously improve our use of language with minimal external input or feedback). That efficiency is whats evidence of intelligence not the fluency itself per se. Further, language is not the basis for intelligence in humans, and the lack of capacity for language in other animals is not evidence for lack of intelligence in other animals.
The capacity for language is not necessarily evidence of intelligence in computer systems, in part because these systems do not share our particular constraints and limitations. Also (and in contrast to other animals) people do not make the same claims about the "intelligence" of protein prediction systems or weather prediction systems or image generation systems that they make about statistical text generation systems (even those these systems are very similar in how they are trained, how they work, what they do).