Longtermism Will Probably Fail
We begin with a look at Daniel Faggella's pro-extinctionism, show how it's nearly identical to longtermism, and then examine a number of dubious assumptions underlying the longtermist ideology.
This article is a bit more technical than usual. I’ll publish some AI bloopers in the next post to make up for it. :-) Thanks for reading, friends!
Acknowledgments: Remmelt Ellen and Kyle Evanoff provided insightful feedback on this piece. They do not necessarily endorse my claims or conclusions.
A Self-Defeating Futurology
You may recall Daniel Faggella from a previous newsletter series. He’s a pro-extinctionist who argues that we should replace humanity with a “worthy successor.” He defines a “worthy successor” as “a posthuman intelligence so capable and morally valuable that you would gladly prefer that it (not humanity) control the government, and determine the future path of life itself.”
The point is to create ever-“higher” lifeforms that expand outward and colonize every corner of the cosmos. On this view, individuals like you and me are nothing more than a means to an end. All current and future beings are fungible and disposable. They are mere “torches” that carry what really matters: life or consciousness, which Faggella dubs the “flame.” The goal is for humanity to create its successor in the form of AGI, which will immediately create its own successor in the form of AGI+, which will immediately create its own successor in the form AGI++, and so on forever — an endless, aimless process that Faggella sees as the ultimate eschatological destiny of Earth-originating “intelligence.”
This is a deeply nihilistic view, which overlooks a number of major problems. I outlined some such problems here, but Faggella has yet to directly respond to them (despite publishing this). A problem I didn’t mention in my previous article is that Faggella’s worldview is built on a contradiction:
He claims that “a safe (or at least not overtly and unnecessarily dangerous) expansion of potentia is the sole moral imperative.” The only goal is to maximize “potentia,” defined as what enables life to persist.
At the same time, he argues that “(in the future) the highest locus of moral value and volition should be alien, inhuman.” He doesn’t want human values to get locked in and determine the future of life. He wants future beings to discover their own radically different set of values.
So, Faggella wants future beings not to be constrained by human values, yet demands that these beings forever embody his value of “expanding potentia.” As I put it in a social media post written for him:
Which is it? Do you want your imperative “maximize potentia” to constrain all future lifeforms, or do you want those lifeforms to discover their own values, which may lead them to antithetical imperatives? If future lifeforms don’t maximize potentia, you’d consider this a great cosmic tragedy — precisely because you’re trying to impose a human value (potentia maximization) on beings you claim should reject human values. That’s why your view is self-defeating: if future lifeforms were to embrace it, they’d reject it.
Faggella’s View Is Just Utilitarianism
Another way to understand Faggella’s eschatological ethics is this: it’s just utilitarianism with a different value theory. Let me explain:
If you were to dissect utilitarian theory, you’d find that it has two components: one deontic and the other axiological. “Deontic” refers to what you ought to do, whereas “axiological” refers to what is good or bad, better or worse. The axiological component says that one state of affairs containing more total “value” is better than another containing less. The deontic component says that you ought to bring about the state with more total value — hence, failing to do this would be morally wrong.
To illustrate: if lying to someone would result in the best possible outcome all things considered (axiological claim), then you should lie (deontic claim). But if telling the truth results in the best outcome, then you should tell the truth. This is what your philosophy professor meant when she said, “Utilitarianism puts the good before the right” — it builds what’s right on what’s good or overall best; it bases the deontic on the axiological.
But what is “good”? What is the “value” that utilitarians say we must maximize? This is where value theory enters the picture: we know what we ought to do — maximize value/the good — but we don’t know what value is without a theory of value to tell us.
The most well-known value theory defended by utilitarians is hedonism, which says that value = pleasurable experiences or happiness. Hence, you can replace “value” in the sentence, “Our sole moral obligation is to maximize value,” with “pleasure” to get: “Our sole moral obligation is to maximize pleasure.” The value theory of hedonism fills in the blank!
What Faggella does is introduce a new value theory, which is even less plausible than hedonism. According to his account, value = potentia (well, it’s really consciousness or life, but potentia is what enables those things to persist and evolve). Hence, “Our sole moral obligation is to maximize potentia.” This is just utilitarianism with a different theory of value.
Utilitarianism Is Just Capitalism
One of the most salient problems with traditional utilitarianism is that it doesn’t care about people as ends in themselves. In fact, it doesn’t care about anything for its own sake except for “pleasure” or “happiness” (if one is a hedonist).
This is why it yields the Repugnant Conclusion, whereby a world full of 1 trillion people with a happiness level of only 1 is better than a world full of 1 million people with a happiness level of 900,000 (if you can even make sense of quantified “happiness levels” — I can’t, but this is how they proceed).
That’s because 1 trillion x 1 units of value = 1 trillion, whereas 1 million x 900,000 = 900 billion. One trillion is more total happiness/value than 900 billion, and hence utilitarians say that we should realize the first world full of basically miserable people over the second world full of wonderful lives if those are our two choices. Utilitarians like William MacAskill, Toby Ord, and Peter Singer don’t see the Repugnant Conclusion as a big deal. They make that claim here.
This is nothing more than capitalism applied to ethics. It’s all about the number-crunching. Morality is a calculation to be solved. Utilitarianism reduces ethics to a branch of economics. People are just replaceable containers of “value,” as they have no value in themselves.
Some bizarre things (aside from the Repugnant Conclusion) follow from this. For example, there is no intrinsic difference between the death of someone and the non-birth of someone else, all else being equal. That is to say, if an unborn possible person would have a happiness level of 100 if they were to exist, then by not existing, the universe is deprived of 100 units of value. We should, therefore, create that person to add 100 units of value to the world. Similarly, if an existing person has a happiness level of 100 but then suddenly dies, the universe is deprived of 100 units of value. In both cases, the outcome is identical: the universe loses 100 units. As Jonathan Bennett once mockingly wrote: “As well as deploring the situation where a person lacks happiness, these philosophers also deplore the situation where some happiness lacks a person.”
You could say something similar about profit in a capitalist system. Not losing $100 million is good. But if there’s another $100 million to gain, you must acquire it. On both the capitalist and utilitarian views, could exist implies should exist: an extra $100 million in profit should be acquired if it could be acquired, and a merely possible person with net-positive value should exist if they could.
These parallels aren’t coincidental. Utilitarianism and industrial capitalism both emerged in the latter 19th century in the very same place: the UK. They are both expressions of an impersonal, quantitative, dehumanizing conception of people as means to an end, and the end as something to be maximized.
Longtermism: Techno-Capitalism Projected into the Stars
I say all of this to introduce the second topic of the article: longtermism, an ideology built upon utilitarian ethics. In particular, it’s founded on the axiological component of utilitarianism, which states that one outcome is better than another if it contains more total value.1
What longtermism adds to utilitarianism is a more expansive, cosmological perspective: cosmologists tell us that the universe is huge and will remain habitable for another 10^100 years or so — a 1 followed by 100 zeros, a much greater amount of time than the universe has so far existed. That means the future population of beings, if we spread throughout the universe, could be enormous. Assuming these beings will bring at least 1 unit of net value into the world during their lifetimes, we therefore have an obligation to realize as many of these future beings as possible — to flood the universe with our descendants until every single nook and cranny, every inch of cosmic real estate, is crammed with people.
Related Article: “Three Lies Longtermists Like to Tell.” Lie #1: Longtermists Care About Avoiding Human Extinction. Lie #2: Longtermists Care About the Long-Term Future. Lie #3: Future People Matter.
Why does it matter that the universe contains the maximum possible amount of value between now and the heat? Well, it just does, because longtermists — and utilitarians — assume that the only appropriate response to value is to maximize it: more, more, more, with no upper limit.
But this ignores a wide range of alternative responses to value such as cherishing, treasuring, revering, caring for, loving, protecting, preserving, sustaining, appreciating, and so on. My approach to ethics emphasizes these alternatives rather than maximization. When I stumble upon a meadow in the woods, I stand there and appreciate it. I take it in and savor the beauty. I don’t think, “This meadow has value. Therefore, the entire world — indeed, the whole universe — should be covered in meadows, because more is always better.” I’m not a capitalist when it comes to ethics.
This is what longtermists think when they look up at the firmament:
Humans Must Be Digitized
To maximize value, we must therefore colonize space. And given that energy sources like stars are burning through their hydrogen reservoirs as we speak, we must do this asap. That’s the central point of Nick Bostrom’s paper “Astronomical Waste.”2 Every second of delay entails some irreversible loss of our “cosmic endowment.” Time is of the essence! You can see why this view appeals to billionaires like Elon Musk.
But how can we colonize space? There’s almost certainly no way for biological humans to do this. Space is incredibly hostile to squishy meat-sacks like us — even colonizing our planetary neighbor, Mars, is likely impossible for human beings, given that Mars has almost no atmosphere, lacks a magnetic field, has much weaker gravity than Earth, and is covered in dirt that’s literally poisonous. The moon is probably the furthest from Earth we’ll ever get.
Hence, humanity will need to become or create digital beings — what I call digital space brains — to colonize the universe in our place. It’s these “posthumans” who will venture beyond our planetary oasis and plunder the cosmos, not us.
More Total Value
As it happens, this has a salutary consequence from a utilitarian point of view: imagine somehow colonizing exoplanets as biological humans. After 20,000 years of travel, we arrive in our spaceships and somehow magically terraform the exoplanet to make it Earth-like and habitable. How many biological humans could then exist on its surface? Maybe 10 billion? Maybe 20 billion?
Now consider an alternative: we arrive as digital space brains and use some magical technology to transform the planet into a giant computer made of computronium (matter optimized to perform computations). We then run a huge virtual reality simulation on this planet-sized computer for digital space brains to reside in. How many of these space brains could exist within the simulation? Longtermists think there could be way more than 20 billion.3 That’s because you can cram more digital people into a given unit of spacetime than you can biological humans. Hence, to maximize value, all future people will need to be digital rather than biological.4
Posthuman Value-Containers
There’s another utilitarian advantage to digitizing “intelligence.” We noted above that, for utilitarians, people have zero intrinsic value. We are the “containers” of value, and the more people/containers there are, the more total value there could be. (That’s why the future population should be as large as possible.)
But here’s the catch: humans can only contain so much value. We’re constrained by our “biological limitations,” and natural selection didn’t optimize us to generate unlimited amounts of goodness. The human value-container is only so deep.
But we could — supposedly — engineer digital posthumans who contain far more value, per individual, than biological humans. We could design them to be much deeper value containers, as illustrated below5:
Do you see where this is going? The ethical foundations of longtermism — utilitarianism — are inherently pro-extinctionist, just like Faggella’s view. To maximize value, we should replace humans with posthumans — and we should do this as soon as possible.
You might think these posthumans would keep some biological humans around, but that wouldn’t be the case: biological humans take up space and use up valuable resources in “suboptimal” ways. There’s no reason for posthumanity to keep us around, and every reason to nudge us out of existence. If the goal is to maximize value in the universe, then our days are numbered.
This is hardly any different from Faggella’s pro-extinctionism. Both aim to maximize something, and in service of maximization they imply that our species should be replaced by a “worthy successor.” The only notable difference is the underlying value theory, though Faggella’s notion of “potentia” also describes a key feature of the longtermist worldview, as maximizing value requires future beings to find new and better ways to ensure the continued survival of life and consciousness.
Six Problems that Longtermists Ignore
But longtermism encounters a number of major problems — nearly all of which its advocates have tended to ignore, dismiss, or wave away. What follows is not an exhaustive list, but rather a collection that I find particularly interesting:
1. Artificial Consciousness?
The entire longtermist worldview is predicated on the possibility of artificial consciousness. Why? Because value seems to depend on consciousness: if digital space brains are “intelligent” but not “conscious,” handing over the universe to them would be like bequeathing it to rocks. Since only digital beings will be able to colonize space, it really matters that artificial consciousness is possible. Literally everything depends on it.
This problem divides into two parts, one metaphysical and the other epistemological.
1.1 The Metaphysics of Mind
Can artificial systems be conscious? Most longtermists assume that’s the case because they think the brain is just a computer — a position called “computational functionalism.” But the “brain as computer” metaphor is just that — a metaphor. Historically, people have metaphorized the brain in terms of whatever the most impressive technology happened to be at the time. After discovering that electricity stimulates nerves, the brain became a telegraph network. When neurons and synapses were identified, it was taken to be a telephone switchboard.6 In the 1940s, cybernetics pushed information feedback loops into the spotlight, and once digital computers arrived, people became convinced that our central nervous system works the same way: neuronal networks are the “wetware” that runs mental algorithms, the software of the brain. In 20 years, it’s entirely possible that the computer metaphor will be replaced by something else.
If the brain isn’t a computer, then artificial consciousness might not be possible. Mind-uploading almost certainly wouldn’t be — i.e., simulating one’s brain on a computer wouldn’t give rise to subjective experiences any more than simulating a hurricane would soak your laptop in rain. There are plenty of reasons for thinking the brain isn’t a computer and that artificial systems can’t be conscious. The neuroscientist Anil Seth provides a useful overview here:
Also, reflect for a moment on what it would take to simulate someone’s brain in silico. Our brains are the most complex objects in the known universe. They consist of ~86 billion neurons and 100 trillion synapses. Waves of electrical activity constantly sweep over them like stormy weather. Signals are spatiotemporally summated in complex ways by interconnected cells. Every second, quivering branches of neurons are rewiring themselves while spurts of blood course through tangles of axons, dendrites, and billions of support cells called glia, whose function we don’t yet fully understand. Your brain is not a static blob, but a constantly wriggling dynamic system squirming about in your skull. Meanwhile, intracellular metabolic processes are producing waste products that accumulate throughout the day and are flushed out at night, when your brain oscillates through different stages of sleep that correlate with distinct wave frequencies.
All of these phenomena would have to be simulated by a computer. The metabolic processes, the brain waves of sleep stages, the quivering axons and dendrites, and perhaps even quantum phenomena that might turn out to be important for cognition and consciousness. Is that technologically feasible? Who knows — maybe not. Given that, for longtermists, everything rests on consciousness being realizable by artificial systems, you’d think they would be obsessing over such questions. But they aren’t.7
1.2 “Testing” for Consciousness
This brings us to the epistemological problem: assuming that artificial consciousness is possible (which I doubt), how could we possibly know that a particular artificial system is actually conscious? What objective “test” could confirm this? (Longtermists would want a test that’s extremely reliable — to the point of near certainty — given how much depends on artificial consciousness.)
If an AI were to say, “Yes, I’m conscious,” could we intersubjectively verify this? AIs, including uploaded minds, could be highly functional zombies: beings with complex outward behaviors but nothing it is like to be them on the inside. Indeed, current LLMs are like this: few people think they’re actually conscious, yet sometimes they’ll tell users that they are.
Perhaps a human being could incrementally upload their mind by replacing one neuron at a time with a silicon chip (a “functional isomorph”) and report back on how the process is affecting their conscious experiences in realtime.8 But what if this person loses consciousness without noticing? What if the functional isomorphs produce the same linguistic output — “I’m conscious” — while the light dims? (As I’ve noted before, there is a closed loop of causes and effects from stimuli impinging upon the eardrum all the way to a vocal response involving the vocal folds, tongue, and lungs. It could be theoretically possible to reproduce this causal loop without consciousness being present.) The philosopher Susan Schneider has proposed a few “tests” to determine if artificial/digital systems are conscious, but in my view they are all inadequate.
This should worry longtermists a lot, which makes it surprising that so few of them talk about it. Imagine that some longtermists create AIs that they mistakenly believe are conscious. They launch these beings into space, where they proliferate in every direction, filling every corner of our future light cone. They take over the universe. But it turns out this founding population wasn’t in fact conscious, and hence our “cosmic endowment” of resources is gobbled up by beings that can’t realize “value.” Whoops! This is why having a robust objective “test” for artificial consciousness — assuming artificial systems can be conscious — is absolutely paramount for longtermism.9
A huge question mark hangs over the entire longtermist project because of these two conundrums.
2. There Won’t Be Digital “People” in the Future
We saw that longtermists argue that there should exist as many digital space brains in the future as possible — an idea I’ve sloganized as, “Won’t someone think of all the digital unborn!” That’s why they’re obsessed with trying to estimate how large the future population could be. Bostrom says there could be 10^38 digital space brains in the Virgo Supercluster, while Toby Newberry claims there could be 10^45 in the Milky Way galaxy. Later, Bostrom estimated 10^58 within the accessible universe as a whole. These estimates are used to motivate their claim that the far future is overwhelmingly important, given that “could exist implies should exist.” It’s not that future people matter more — it’s just a numbers game: there could be far more of them than the total number of humans who currently exist.
The problem is that such calculations assume there will be individual people who can be counted — that future people will be individuatable. But why think this will be the case? If future minds take the form of pure information, of computer algorithms, then they could — and no doubt would — share memories, personality traits, experiences, preferences, beliefs, desires, and so on, with the metaphorical click of a button.10 The boundaries between “individuals” would become porous, and the world would become a digital soup of ontological chaos. Perhaps these digital minds would aggregate into some kind of amorphous wriggling hive mind, comprised of jostling mental fragments constantly recombining with each other in a phantasmagoria of incessant change. Who’s to say?
However things turn out, we can be sure that the digital world will be profoundly different than the world we live in today. It may very well mark a completely new post-individual era in which there are no countable “people” at all. And if there are no countable people, then the estimates that longtermists rely on to motivate their view are completely meaningless.
3. Digital Prozac
Even if digital space brains were to somehow maintain their individuality, longtermists still have to explain why any of these space brains would enjoy living in giant computer simulations in the first place. What would give their lives meaning and purpose? Why on Earth would digital posthumans be happy in vast simulated realities crammed with trillions and trillions of other “disembodied” minds?
I have never once seen any longtermists address this question. Perhaps there would be digital pharmacies to provide digital people with digital Prozac to get through? Or perhaps future space brains will see it as their “moral duty” to somehow force themselves to be “happy” for the sake of maximizing value — because they understand they’re means to an end, according to the capitalist ethics of utilitarianism.
Imagine for a moment what life within a giant simulation might be like. The laws of physics as we know them need not apply: one could design virtual reality worlds in which there’s no gravity, the fundamental particles and forces are different, and F ≠ ma. The possibilities are terrifyingly endless. Would everyone have their own virtual reality world? Would a bunch of digital space brains share a world? If so, who would get to design it? What if it were hacked and everything suddenly changed? What if one’s friends were permanently deleted? Who would maintain law and order? How would people prevent attacks from outside their little Matrix, originating in the physical (rather than digital) universe?
Such worries bring up two additional issues:
4. Immortality, Bunkers, and Eternal Torture
We start with immortality. Digital space brains would be immortal, because software is immortal. As long as the hardware exists to realize the software, digital posthumans could live until the heat death of the universe.
But consider the implications: if you were a digital space brain and were thinking rationally, you would do literally nothing else every day — for trillions of years — but obsess over your survival. Why? The standard “expected value” definition of risk is “probability multiplied by the consequences.” This means that a negative outcome with a tiny probability of occurring could still constitute an enormous risk if the consequences are sufficiently high. If you’re functionally immortal, then the consequences of dying would be astronomically huge, because you’d be losing trillions and trillions of future years of life.
To illustrate, if I were to get hit by a bus tomorrow, I’d lose perhaps 4 decades of my natural life. But if I could live for the next 1 billion years, then getting hit by a bus would be orders of magnitude worse. Consequently, the risk of getting hit by a bus — even if it remains extremely improbable on any given day — would skyrocket. Insofar as I’m rational, I’d never cross a road again. Indeed, since there are many possible but improbable ways of dying, I should just build a bunker and never come out.
The same logic applies to functionally immortal digital posthumans. Living in their computer simulations, they should take every imaginable precaution to ensure their continued survival, given that bad outcomes with vanishingly small probabilities would pose massive risks. They should fortify the planetary computers on which their simulations run with all sorts of exotic weaponry just in case an alien species of posthumans outside the Matrix were to attack. Individually — if they have individuality — they should constantly back up their digital minds in case something happens, which would surely foment fights between digital people over limited server space. Every moment spent not ensuring one’s continued survival would be irrational, given the enormous stakes of dying. This brings us to the second issue:
5. The Anarchic Cosmopolitical Arena: Constant Catastrophic Wars
Why do longtermists assume that space won’t become a theater of constant, catastrophic wars? Due to the vastitude of the cosmos and the hard limit of the speed of light, there’s no possibility of a well-coordinated cosmic Leviathan to provide security. Mutually assured destruction (MAD) won’t work either, and populations of digital posthumans would gradually evolve radically different capacities, traditions, abilities, languages, belief systems, technologies, scientific theories, etc., thus making mutual trust impossible.
What, then, is left to keep the peace? Each solar civilization, or archipelago of civilizations, will find itself in a Hobbesian trap in which preemptive violence becomes the only rational option to guarantee its survival. Spirals of militarization (the “security dilemma”) will greatly exacerbate the situation, as defensive weapons are misinterpreted as offensive threats.
I’ve written about this at length elsewhere, so won’t elaborate here. Suffice it to say that the anarchic nature of the cosmopolitical realm would be primed for constant conflict, and exotic space weapons could potentially obliterate neighboring civilizations at the speed of light. Of course there will be devastating wars all the time — no matter how irenic our posthuman progeny are, they won’t be able to escape the death traps of game theory.
6. Longtermists Face Pascal’s Wager, Not Pascal’s Mugging
The problem of immortality, illustrated above, is an example of what Eliezer Yudkowsky and Bostrom call “Pascal’s mugging.” This occurs when tiny probabilities are paired with enormous stakes. Let’s say that there could be 10^58 digital space brains between now and the heat death, as Bostrom argues. That means the cost of undergoing final extinction (see below) is HUGE. It follows that even if there’s only a teeny-tiny probability of extinction happening, the “expected value” of preventing this scenario will still be massive. We should, therefore, redirect nearly all of our finite resources away from non-extinction risks in favor of projects aimed at ensuring the existence of future digital space brains.
Pascal’s mugging points to extreme cases where expected value theory seems to break down. When tiny probabilities and enormous stakes are involved, expected value theory recommends fanatical options.
Many longtermists claim to oppose “human extinction.” But there are two linguistic tricks they play in making such claims. The first is to redefine “human” so that it encompasses our posthuman successors. On this “Broad Definition,” defended by the likes of MacAskill, Ord, Bostrom, Greaves, Beckstead, and others, our species could die out next year without “human extinction” having occurred — so long as posthumanity replaces us, “humanity” will endure. Second, when they talk about “extinction,” they’re referring to final rather than terminal extinction. Final extinction happens when our species disappears and we leave behind no successors, whereas terminal extinction just refers to our species disappearing. Longtermists don’t actually care about avoiding terminal extinction — they care about avoiding final extinction. That’s it. (It’s with respect to terminal extinction that longtermism is a pro-extinctionist view like Faggella’s.)
Always be wary of the language games they play: when they talk about the importance of preventing “human extinction,” they’re using that term in a very particular, idiosyncratic, nonstandard way. In truth, they don’t actually care about the long-term survival of our species.
The problem is that if you ask a cosmologist whether the heat death of the universe would mark a definite end to information processing, life, intelligence, etc., they will answer “no.” Although speculative, it’s possible that future digital people could find a way to escape the heat death — e.g., by tunneling into a neighboring universe, creating their own universe from scratch, or utilizing some strange futuristic physics to intervene and prevent the heat death from occurring.
Consequently, there could be an infinite number of future digital people. Perhaps our distant descendants tunnel into a neighboring universe, and then iterate this process each time their new cosmic abode collapses into thermodynamic equilibrium. By endlessly iterating the process, they could exist forever, thus generating infinite amounts of value. Improbable? Yes. Impossible? No: there’s a nonzero chance of this happening.
Longtermists should thus use this infinitely higher estimate of future people in their expected value calculations. They want to know what the stakes are, and it appears the stakes could be infinite.
Doing this, however, replaces Pascal’s mugging with Pascal’s wager. The former involves huge stakes paired with tiny probabilities, whereas the latter involves infinite stakes. The conclusion is that the only thing we ever should spend our money and resources on is preventing (final) extinction. Final-extinction risk mitigation wins every time without exception. This is a direct consequence of longtermism plus the cosmological fact that there could be infinite people in the future.
If longtermists were intellectually serious, they would acknowledge that their view yields the most extreme form of “fanaticism” imaginable: literally nothing matters except continuing to survive so that infinite amounts of value can be realized in the future. It’s Pascal’s wager applied to cosmic rather than personal eschatology.
Conclusion
Longtermism proposes a very bizarre futurology. This futurology is based on a number of assumptions, almost none of which are seriously examined by longtermists. The point of this article wasn’t primarily to critique the longtermist ideology, but to highlight a small handful of its underlying assumptions.11 It behooves longtermists to address such issues, but given the cult-like nature of the longtermist community, I doubt they will.
The article also aimed to show how two supposedly different ideologies — Faggella’s “worthy successor” view and longtermism — are nearly identical. Faggella is just a longtermist with an idiosyncratic theory of value, while longtermism has pro-extinctionist implications no less than Faggella’s eschatology.
As I’ve argued, many of these TESCREAL or TESCREAL-adjacent views are far more similar than they are different, including longtermism, effective accelerationism, cosmism, singularitarianism, and the pro-extinctionism of people like Larry Page, Beff Jezos, and Eliezer Yudkowsky. There are differences, but they are minor. All must be stopped insofar as we want a good, flourishing future for our species.
As always:
Thanks for reading and I’ll see you on the other side!
The deontic component, once again, then says that realizing this better outcome is morally obligatory — it’s what you ought to do.
This article was intended to support the longtermist view and the claim that mitigating existential risks should be our top four global priorities, but in fact it provides an excellent reductio of longtermism.
This is why their estimates of digital people are always higher than their estimates of biological people.
After reading this above-paragraph, my friend Kyle Evanoff offered some insightful comments, which I’m paraphrasing and elaborating here:
We don’t actually know that there could be more digital than biological people in the future, because we haven’t invented any digital space brains yet. We don’t know what kinds of technologies will be necessary for them to exist, how much space they’ll occupy, and how much energy they’ll consume. AI data centers take up 100,000 to 10 million square feet, whereas the human brain fits inside a single skull; our brains use about 20 Watts of power, whereas current AI systems suck up perhaps 1 or 2 billion Watts to perform similar cognitive tasks. Perhaps AI won’t get much more energy efficient in the future, and perhaps simulating a human brain will require massive megastructures like AI data centers.
Longtermists would counter that energy isn’t a concern because our digital descendants will just build Dyson spheres around stars to power their “planet-sized” computers. But Dyson spheres are also highly speculative, and quite possibly infeasible to build around single stars. At the very least, they would require disassembling an entire planet the size of Jupiter.
So, perhaps there will be far fewer digital people in the future than longtermists anticipate, and thus much less “value.” If longtermism were a futurology worth taking seriously, you’d see longtermists addressing these issues. Instead, they feel entitled to ignore these questions because they assume that “utopian” technology and AI superintelligences will “magically” solve all such problems.
To be clear, this is once again mere speculation. We have no idea if this is possible.
I’m paraphrasing Matthew Cobb, who writes: “With the discovery that nerves respond to electrical stimulation, in the nineteenth century the brain was seen first as some kind of telegraph network and then, following the identification of neurons and synapses, as a telephone exchange, allowing for flexible organisation and output (this metaphor is still occasionally used in research articles).”
Furthermore, there’s a second issue that I brought up in a previous article, namely, that mind-uploading is not the same as self-uploading. Even if mind-uploading were possible, self-uploading doesn’t appear to be. Many longtermists are hoping to join the hoards of digital posthumans flitting about the universe — but that’s not going to work. Once more, longtermists consistently ignore this problem.
Note that this process would be impossible. The point is to continue replacing parts of the biological brain with neural chips until the entire brain is artificial, at which point one’s mind would be fully uploaded. During the process, one would repeatedly introspect to see if the chips are influencing consciousness. The problem is the biological brain is supported by vasculature and other systems that would be damaged in the process, causing the individual to die.
Longtermists do discuss artificial consciousness for a different reason: if AIs are or become conscious, then they would constitute moral patients who, as such, ought to be treated in a certain way. But I’ve never seen any longtermist worrying about how the metaphysical or epistemological problems here outlined could render their techno-eschatology infeasible.
A digital, many-person form of craniopagus.
I’m not sure I did a great job of outlining these assumptions. This article was written rather hastily. If longtermists respond, they should focus on steelmanned versions of what I present.
brilliant thank you for putting in the effort to dissect this