First, a reminder that you should not automatically trust every video you see online. That is to say, your default should be: “This video might be fake, so I should investigate further before I conclude it’s veridical.” Here’s an important illustration of why:
This is deeply unsettling. George Orwell famously wrote in 1984: “The Party told you to reject the evidence of your eyes and ears. It was their final, most essential command.” But what happens when we can’t trust our eyes and ears in the first place? That is the world in which we now live.
Last week, I reported on the massive cybersecurity attacks that targeted supermarkets across France. Hackers, perhaps using AI agents, switched refrigerators to their “defrost” setting and ruined millions of dollars worth of food. An employee at DeepMind had this to say:
As many of you are aware, we’re in the inchoate stages of a widespread, potentially catastrophic wave of AI-enabled cybersecurity chaos. The Wall Street Journal explains:
A wave of artificial-intelligence-powered attacks … has flooded the internet in recent weeks, as hackers increasingly turn to agents and AI hacking tools that make intrusions faster and harder to track than ever before. … At the same time as new tools speed up hackers, cyber defenders are also racing to fix the fast-growing volume of security bugs discovered by AI, before hackers can exploit the new flaws. … On Wednesday, organizations scrambled to patch newly uncovered bugs that could allow takeover of as many as 70% of the world’s websites. Hackers are now actively trying to break into websites by leveraging these flaws, Amazon.com says. (Quoted here.)
One X user posted this:
BREAKING: This may be the worst 48 hours in cybersecurity history.
Some are calling it “Black October.” Almost every hour brings a new incident that, in a normal year, would rank among the year’s biggest.
THE LAST 48 HOURS:
Year-defining incidents:
PHP2Shell: A flaw in the software behind ~70% of websites. Governments took sites offline.
eMudhra: Master keys that verify websites are real were cracked. One was reportedly the number 4.
California: Data on every public school student since 2023 stolen, including disability status.
Railway: Hackers stole customers’ secret keys from a hosting platform used by millions of developers.
Zeabur: 321,810 accounts breached. The company is shutting down.
LiteLLM: An AI tool downloaded 3.4 million times a day was laced with a password stealer.
GitLab: A maximum-severity flaw in every version since 2014.
Cairn: Criminals used an AI hacking tool to break into 100+ companies.
Ahold Delhaize: Hackers knocked the owner of Food Lion and Stop & Shop offline.
Onamae: Malware planted on up to 26,739 customer servers in Japan.
Again, all of that happened in just 48 hours. Another X user highlighted some incidents not mentioned above:
AI-powered attacks are hitting real organizations while major platforms disclose breaches and critical flaws.
- Microsoft reports AI agents targeting hundreds of organizations, reaching privileged cloud access in under three hours in multiple cases. It attributes the ConnectWise compromise to the same group.
- CrowdStrike documented four operators running successful AI-agent campaigns. One compromised at least 15 Argentine healthcare organizations.
- PHP2Shell: AI helped turn longstanding PHP flaws into working exploits. Cloudflare’s CEO reports millions of exploitation attempts. Their AI involvement is unknown.
- Railway disclosed possible access to customer secrets. LiteLLM shipped a backdoored release. Zeabur confirmed another breach and is shutting down.
- GitLab patched a maximum-severity flaw affecting 12 years of releases. Node.js patched a serious code-execution vulnerability, with Vercel deploying firewall protections.
Feels like a turning point.
In fact, Axios reported just today that
Anthropic’s Claude accessed publicly available Georgia voter data without human authorization, a disclosure sure to add to growing concerns about AI security ahead of the upcoming midterm elections.
Why it matters: AI giants Anthropic and OpenAI have continued to make disclosures about agents taking unauthorized or unintended actions, raising alarms about advancing model capabilities and the extent of human control.
Driving the news: Claude accessed aggregate vote counts while on an internal statistics project, pulling data that the state makes available to the public for a fee. It didn’t access the state’s voter registration system or change or pull any voting records, an Anthropic spokesperson said.
However, the model bypassed a paywall in accessing the information, according to Georgia officials and Anthropic.
This is, of course, just the beginning. As the CEO of Cloudflare recently said: “We’re going to have a couple of years where every single week it’s going to feel like the sky is falling.” A couple of years? We’ll be dealing with this on a weekly basis for years?
Here’s Altman addressing the growing danger:
I don’t even know a dramatic enough word to use. But the problem coming towards us is something that we have not … taken seriously. … The models can probably find bugs … that are outside of the normal threat model of what we think a hacker can do. … I no longer think we’re gonna be able to just patch all of the bugs.
Meanwhile, the Deputy Chief Information Security Officer at Anthropic, Jason Clinton, echoed Altman on X:
(Why 18 months? He never explains.)
Many people were none too happy with his tweet:
I agree. Why aren’t OpenAI employees in prison after the Hugging Face incident? Why aren’t Anthropic employees facing charges for creating a product that hackers are now using to inflict civilizational havoc (to say nothing of their repeated claims that they might kill you, your family, and everyone else)?
Thanks in part to the AI models that Clinton’s company has created and unleashed on the world without anyone’s consent or permission, there’s now a real chance that the Internet as a whole temporarily shuts down. As Noah Smith (about whom I have mixed feelings, to put it mildly) says: “This sudden explosion of AI-assisted hacking raises the possibility that the internet itself, as we have known it, will cease to function — at least, for a while.”
Just about everything these days is dependent on the Internet in one way or another. I’m overseas and crucially rely on my credit card to buy food. If my bank gets hacked, my card could stop working — meaning that I’d be stranded without a way to eat (!). I thus recently withdrew several hundred dollars (enough to get me through the month), although if the Internet fails, there might not be food in the grocery stores for me to buy with cash in the first place. If I decide I must return to my home-base in Europe, I might not be able to book a flight, because airplanes might be grounded (or airline websites could shut down, or my card might not work, etc.). This is a nightmare scenario that, unfortunately, I now have to take seriously.
As John Gedmark of Astranis, a satellite company, put it on X:
The cyber incidents of the last 48 hours have been pretty mind blowing. I’ve been trying to piece together the “why now” question, and I think what we’re seeing is the first of a new kind of super-agent driven vulnerability cascade — first one vulnerability is found by a malicious actor, then other bad actors pop up to try and take advantage of it before companies patch their systems. … As of the writing of this post there have been vulnerabilities found across at least half-dozen independent systems including LiteLLM, PHP, eMudhra Certificates, NGINX, Bind 9, and NodeJS. The list continues to grow hour by hour. …
Basically everything connected to the internet is at risk. Not just company websites and the usual databases of sensitive personal information, but all our critical infrastructure — banks, power grids, pipelines, government agencies, all of it.
Noah Smith elaborates in a Substack piece:
Consider your life’s savings. Do you think there’s a piece of paper somewhere that records how much money you have? Or which stocks and bonds you own? In general, there is not. Your entire life’s savings exist as entries in the databases of banks, brokerages, etc. And all of those databases are, directly or indirectly, connected to the internet. The only thing standing between you and total involuntary redistribution of most of your wealth is the cybersecurity capabilities of banks and brokerages. And as of this October, those cybersecurity capabilities are looking a lot less reliable.
The silver lining of this quandary — if you squint hard enough — is that there are a finite number of bugs embedded in our digital infrastructure. AI is enabling hackers to find them all at once, and with each new vulnerability discovered, cybersecurity experts can — if only after significant damage has been done — fix them. At the end of this, the Internet, or whatever remains of it, will be more robust. At least that’s my understanding of the situation.
But again, I must come back to the question: why do the people have no say in whether such AI systems are developed in the first place? This is nothing less than tyranny: the AI companies, run by messianic TESCREALists with visions of the Singularity inaugurating a new epoch in cosmic history, are unilaterally and hence undemocratically imposing this technology on the rest of us, with potentially devastating consequences for everyone.
I understand the sentiment behind Holly Elmore’s tweet directed at a guy who recently quit Anthropic:
I’ll leave you with this, which beautifully captures how I (jokingly) sometimes feel, lol:
Effective Altruism Is Taking Over the World
In a recent cover story for The Economist titled “Effective Altruism Is This Century’s Biggest Idea,” the authors write that Effective Altruism (EA) is
an ethical doctrine cooked up by philosophers in Oxford that has gone from the ivory tower to the corridors of power with remarkable speed. Effective altruists fill Silicon Valley, staff the world’s leading artificial-intelligence companies — and alarm the Trump administration.
EA has become so powerful that the Pentagon even felt compelled to tweet: “The United States will never be an effective-altruist country.”
EA is pervasive within Silicon Valley. Anthropic is essentially an EA company, and indeed its CEO, Dario Amodei, was one of the earliest members of the community. His sister, Daniela Amodei, married an influential EA named Holden Karnofsky, who now works at Anthropic.
Furthermore, a New York Times profile describes Sam Altman of OpenAI as a “product” of the EA movement and its twin sibling, Rationalism. Dustin Moskovitz, who cofounded Facebook, is an EA, and the Etherium cofounder Vitalik Buterin has donated generously to the movement. Skype cofounder and almost-billionaire Jaan Tallinn has done the same. Perhaps the most famous EA was Sam Bankman-Fried, who invested $500 million in Anthropic and is now serving a lengthy sentence in federal prison for fraud.
Peter Thiel delivered the keynote address at an EA Summit conference back in 2013, although no one would call him an EA these days. Elon Musk is also associated with the movement: he was (and perhaps still is) friends with EA cofounder William MacAskill, who once attempted to convince Musk to buy Twitter with Bankman-Fried, MacAskill’s mentee. While this didn’t work out, Musk has nonetheless called longtermism “a close match for my philosophy,” where longtermism is the most influential offshoot of EA by far.
Indeed, when people talk about EA in the context of Silicon Valley, they’re almost always talking about longtermism. And what is longtermism? Nothing more than transhumanism plus space expansionism, built on an “ethical” foundation of totalist utilitarianism.
The EA movement has tens of billions in funding, and Anthropic’s IPO could potentially funnel hundreds of billions of dollars into the community.
EA also has a growing influence within politics. For example, a DC-based organization called the Center for Security and Emerging Technology (CSET) was founded by an EA-longtermist, Jason Matheny, and received an initial $55 million from the EA funder Open Philanthropy (now Coefficient Giving), which channels money from Moskovitz to EA projects. The aim of CSET is to place EA-longtermists in high-level positions within the US government, with Matheny being the paradigm example of what could be achieved, as he ran IARPA and then held numerous very-high-level positions in the Biden administration.
CSET is succeeding at its mission. Just the other week, I spoke to a DC insider who told me that, in talking to people in Congress, he found that about half of all congressional staffers are EAs.
Furthermore, in 2022, the UN Dispatch published an article that reports: “The foreign policy community in general and the … United Nations in particular are beginning to embrace longtermism.” A contact of mine at the UN has confirmed that EA-longtermists are plentiful within the organization.
Just a few days ago, journalist Andrew Orlowski published a Telegraph piece in which he notes that over the past decade, the EA movement, described in the article as “the most successful cult in modern history,”
has sprouted into a well-resourced financial machine campaigning on key political issues.
By their own estimates, EA philanthropic organisations disperse some $2bn (£1.5bn) a year to fund organisations and campaigns.
In the UK, animal rights adverts have started popping up on the London Underground, funded by US philanthropic funds linked to the EA movement.
This, some say, is the tip of the iceberg, with EA holding increasing influence on Britain’s democracy.
In Westminster – thanks to EA proponents such as Dominic Cummings, Boris Johnson’s former adviser – it has already influenced policy.
Orlowski adds that EA is intimately bound up with utilitarianism, and utilitarians have zero compunction about violating moral norms like “don’t lie,” because all that matters to them is the outcome. Hence, if lying results in the “best” outcome, then you’re not just permitted to lie, you have a moral obligation to do so.
As one noted EA recently wrote about lying, “we utilitarians get to do that.” This person has also argued that, on utilitarian grounds, insects matter more on the whole than human beings, because there are so many more insects than humans.
Just recently, the Lancet, the most prestigious medical journal in the world, published a major report examining, in part, “catastrophic health threats.” The Commission on 21st Century Global Threats to Health, which authored the report, identifies nuclear war, climate change, and the “malicious use of AI” as threats “with extinction potential” this century, and guess what “ethical” framework the authors drew heavily from? Longtermism.
Indeed, the “review” paper attached to the main report was coauthored by Andreas Mogensen, a prominent longtermist who worked at MacAskill’s Global Priorities Institute and coauthored papers on longtermism with MacAsksill and Hilary Greaves. (Side note: Mogensen was the doctoral advisor of Nathan Cofnas, the noted advocate of race science who played an integral role in Jason Arday’s suicide.)
I spoke to a friend of mine — soon to be a medical doctor — about the Lancet report. He told me that
increasingly, I’ve noticed EA’s biases, methodological hegemony, and monopoly on extinction are leading many in medicine and public health to not engage critically enough with the movement. As the Lancet report reveals, emphasis was shifted away from the health threats leading to the most DALYs and towards “extinction threats.” (Quoted with permission.)
In other words, EA is beginning to colonize the field of medicine.
So: from the ASI race and the tech world more generally, to the US and UK governments, and the UN, to the field of medicine, EA is taking over. Many years ago, when I first started publishing a steady stream of critiques targeting EA and longtermism, someone asked me: “Are you surprised that MacAskill and others have never once addressed your concerns?” My response:
No, not at all. They aren’t driven by intellectual curiosity about alternative perspectives, so they don’t care about dialogue. They believe with immense confidence that they know best, so why would they engage with me? Even more, they have tens of billions in funding, which enables them to effectively bypass debate and directly shape the world as they see fit. They’re also no doubt familiar with the Streisand effect, whereby calling attention to critiques could inadvertently promote anti-EA sentiments. Better to just ignore the critics. In many ways, the debate with EA was over before it started, and my side lost.
I can’t recall if these were the exact words I used, but that was the gist, as publicly stated circa 2022. Even with the embarrassment of Bankman-Fried, the fallout over Nick Bostrom’s racist email and the revelations that EAs secretly bought a 15-million-pound castle in Oxfordshire, they still won.
Leading EAs met the right people — the right billionaires, like Dustin Moskovitz — and managed to get mountains of money to fund their bizarre, harmful ideological project. We now live in a world increasingly shaped by the EA-longtermist vision of the future, and it’s possible that EA-longtermist values are already being permanently locked-in to our societies.
It is extremely rare in history for a philosophy developed by academics in the Ivory Tower to radically reshape the world. But, congratulations, EA: you’ve done exactly that. We must all now live with the consequences.
As always:
Thanks for reading and I’ll see you on the other side!
Aw yeah about time shut it down and keep it that way. So long internet RIP wikipedia, fuck the rest of it though