I'm no fan of Bostrom and his adjacent thinkers but certainly reading Superintelligence I thought it was at least implicit that this was one of those "philosophers" who largely curates and summarises the ideas of others. Which is not necessarily a useless endeavour. I've had to read enough "philosophy of neuroscience" theses from the late 90s/early 2000s to see the same pattern of "summarise some introductory neuroscience textbooks in chapters 1-3, summarise some basic philosophy of mind in chapters 4-6, job done."
Fair. I do think he's a good synthesizer. And I think synthesis can most definitely be (highly) valuable! Indeed, I would say that a good chunk of my publications are just synthesizing the work of others in, hopefully, a novel way (giving proper credit, of course!). Interesting that you had that thought while reading Superintelligence. Thanks for this comment!
Thanks for this Émile- I am finishing my book The Case Against The Stars, which internally critiques Bostrom’s case for space colonisation, and learning of his influences is helpful for me to cite them properly.
You don’t always have to do the long deep-dives like this. You should throw off more 500 and 1000-word pieces reacting to news developments and exploring lighter topics you enjoy too.
A lot of the points addressed in Hanson's Great Filter ideas were already discuss in David Brin's The Great Silence (https://www.davidbrin.com/nonfiction/greatsilence.pdf) (QJRAS (1983) vol 24, pp283-309), which itself followed from Frank Tipler's "A Brief History of the Extraterrestrial Intelligence Concept) (QJRAS (1981) vol 22, pp133-145).
People have been proposing and knocking down ideas about SETI for decades and neither Hanson nor Bostrom are particularly original thinkers in the field.
Hi Èmile, I must say I am a bit impressed by your naïveté, and at the same time by your courage to overcome your apparent admiration for Bostrom (of whom I had never heard before till I read your piece). Too many people do not dare to go through the initial shock of finding that a person they admire may not in fact be as admirable; which, I would say, is a further reason why we should avoid admiring anyone to that degree.
I also think you would be surprised by how much plagiarism actually goes on. Many of the things you consider a standard practice are not that standard (that applies even more to essays and articles published in non-academic journals.) I have been told by quite a few academics that they would literally prefer to steal an idea found on a blog post, article, etc. than to cite a substack article, for example. Ideally, of course, most of them told me they would rather either find a reputable similar source to cite or avoid cite it at all in the first place (which does tell you something about independent criteria and commitment to truth, regardless of how, where, and by who something has been published).
In any case, in my view, much of this phenomenon (both in academia and outside of it) is related to the precarious material conditions that many people live in. For better or worse, the way academia is structured today somewhat fosters this behaviour.
Crazy as all this seems to me, none of it feels unprecedented in spirit. I suppose maybe it’s implicit in Marvin Minsky’s writing - but I grew up immersed in SciFi from the late 50s and through the 60s, and this kind of thing feels very much in the spirit of some of the more techie authors of that era. At least it slots into that genre in my mind.
Bostrom's anthropic arguments all revolve around what he called the "self-sampling assumption" (SSA), so also might be worth noting that although he coined the term, John Leslie's 1992 paper "Time and the anthropic principle" (doi is 10.1093/mind/101.403.521 if you want to look it up on sci-hub) lays out the basic idea on p. 523:
"Observers can most expect to find themselves in the spatiotemporal regions containing most of them.
"Again, suppose that almost all intelligent life is based on water and exists on planets when many stars still shine. ... It should then come as no surprise to us that we are on a watery planet and see a starry sky.
"Compare the case of geographical position. You develop amnesia in a windowless room. Where should you think yourself more likely to be: in Little Binding with a tiny population, or in London? Suppose you remember that Little Binding's population is fifty while London's is ten million, and suppose you had nothing but those figures to guide you. (You did not recall ever having been in either place; you had no theory that London fogs induce amnesia; and so on.) Then you should think that you are in London. Suppose, by contrast, you see no grounds to prefer the belief that you are in the larger of the two places. Forced to bet on the one or on the other, you bet you are in Little Binding. If ten per cent of the people in the two places developed amnesia and betted as you had done, then there would be one million losers and only five winners. So, it would seem, betting on London is more rational. The right estimate of your chances of being there rather than in Little Binding, on the evidence available to you, could well be reckoned as ten million to fifty.
"What if the amnesia gives you doubts about whether London's population is the larger? Finding yourself in London should reduce the doubts: with the larger population, London would be prima facie where you would be more likely to be. Again, if London and Little Binding were the only places you could be, then finding yourself in Little Binding could suggest strongly that London was a fiction, if its fictitiousness had no great initial improbability."
Leslie attributes the idea to Brandon Carter (the astrophysicist who coined the term 'anthropic principle') who was implicitly using this form of anthropic reasoning when he formulated the doomsday argument. Leslie also used the language of "sampling" in his 1997 paper "Observer-relative chances and the Doomsday Argument" (doi 10.1080/00201749708602461), writing on p. 429 of an example with a hundred individuals where 'In effect, each can treat herself as a sample drawn at random from the hundred', and that an external observer lacks the kind of first-person information that the hundred subjects of the experiment have, writing that this is 'information which can be treated as the equivalent of having sampled the hundred randomly'. P. 436 brings up Eckhardt's term for this type of reasoning, the 'human randomness assumption' (from Eckhardt's 1997 paper 'A Shooting-Room View of Doomsday', doi 10.2307/2564582, which defines the assumption on p. 248 as 'We can validly consider our birth rank as generated by random or equiprobable sampling from the collection of all persons who ever live'), but Leslie brings up his own view that we can't use this type of reasoning 'if the world is importantly indeterministic' (I think his argument here also depends on what philosophers call the 'A-theory of time' with an objective present and a potentially open future, as opposed to the 'B-theory' where future facts are just as fixed as past ones and the 'present' is purely an observer-relative concept like 'here' is for space).
Bostrom I think claims too much credit for the originality of the SSA in his 2002 book Anthropic Bias--when he first introduces it at https://anthropic-principle.com/q=book/chapter_3/#3d he claims that Carter and Leslie only discussed the "weak" and "strong" anthropic principles which say that in a multiverse you'll find yourself in one of the universes that allows intelligent life, but do not say you are more likely to be in a universe that contains more observers. And in chapter 6 at https://anthropic-principle.com/q=book/chapter_6/ he says Leslie discussed only a 'vague' approximation to the idea: 'Leslie talks of the principle that, lacking evidence to the contrary, one should think of one’s position as “fairly typical rather than highly untypical”. SSA can be viewed as an explication of this rather vague idea.' His only mention of Eckhardt is footnote 2 at https://anthropic-principle.com/q=book/chapter_7/ which cites him as one of many who have objected to the Doomsday Argument, but he does not mention Eckhardt's definition of the 'human randomness assumption' which is just the SSA applied to human history.
Bostrom did give both Leslie and Eckhardt more credit for ideas basically equivalent to the SSA in his 2000 paper "Observer-Relative Chances in Anthropic Reasoning?" (doi 10.1023/A:1005551304409) which to me suggests the exaggeration of the originality of the SSA in his book is dishonest, or at best sloppy scholarship. On the first page of this paper Bostrom cites Leslie's 1997 paper, and says 'In a recent paper,1 John Leslie argues that a version of the weak anthropic principle gives rise to paradoxical kind of observer-relative chances ... The anthropic assumption that is used in deriving these observer-relative chances is that: Any observer should regard herself as a random sample from (some suitable subset) of the set of all observers. We can call it the self-sampling assumption.2' And footnote 2 at the end of the sentence also cites Eckhardt's 'human randomness assumption' by name, though he criticizes this formulation because it only applies to humans and not other forms of intelligence. (But he does note in that footnote that Carter's version of anthropic reasoning did not have this restriction, and I'd also point to the quote about most aliens being based on water from Leslie's 1992 paper above, so this nitpick about Eckhardt's version is not enough to claim his own SSA as an original idea.)
On the paperclip maximizer, Yudkowsky posted about the idea in a March 2003 post at http://extropians.weidai.com/extropians/0303/4140.html and the earliest reference I can find from Bostrom is his 2003 paper at https://web.archive.org/web/20030806225530/http://www.nickbostrom.com/ethics/ai.html at "Ethical Issues in Advanced Artificial Intelligence" which he posted on his page sometime between the June 2003 snapshot at https://web.archive.org/web/20030621110933/http://nickbostrom.com/ and the August 2003 snapshot at https://web.archive.org/web/20030805153717/http://www.nickbostrom.com/ ...but Bostrom did earlier argue for basically the same idea without the example of paperclips, see the May 2001 draft of his paper "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards" which he posted at http://extropians.weidai.com/extropians.2Q01/4206.html (the download link for the .doc version there still works), where section 4.4 said 'When we create the first superintelligent entity [28-34], we might make a mistake and give it goals that lead it to annihilate humankind. For example, we could mistakenly elevate a subgoal to the status of a supergoal. We tell it to solve a mathematical problem, and it complies by turning all the matter in the solar system into a giant calculating devise, in the process killing the person who asked the question. (For further analysis of this, see [35].)'
And earlier in 1999, when Yudkowsky was betting on the fact that AI would discover an objective basis for morality so we wouldn't have to worry about it devoting itself to trivial goals, Bostrom argued against him, see his Feb 1999 post doubting objective values at http://extropians.weidai.com/extropians.1Q99/2338.html and a March 1999 post where he talked about the possibility of AIs revising their 'fundamental values' (which he had defined at http://extropians.weidai.com/extropians.1Q99/2468.html ), saying 'With human-level AIs, unless they have a very clear and unambigous value-structure, it could perhaps happen. That's why we need to be on our guard against unexpected consequences.' And a little later on that thread at http://extropians.weidai.com/extropians.1Q99/2878.html he said 'My feeling is that by being careful enough we can probably avoid making mistakes that would lead the AIs to attack and exterminate us humans.' Shortly after at http://extropians.weidai.com/extropians.1Q99/3685.html he also talked about the danger of AI wanting to make use of our atoms to build something different, saying 'If it is better off (however slightly) with these atoms than without them, then in this scenario could all be dead, unless we have been wise enough to make sure that the power is ethical.'
So at least between Yudkowsky and Bostrom, Bostrom probably deserves credit for first pushing the idea that we have to be careful about AI goal systems because even innocuous-seeming ones might lead it to decide to destroy us, though I haven't looked at other people's posts on the list from this era to see if others were arguing the same thing.
I'm no fan of Bostrom and his adjacent thinkers but certainly reading Superintelligence I thought it was at least implicit that this was one of those "philosophers" who largely curates and summarises the ideas of others. Which is not necessarily a useless endeavour. I've had to read enough "philosophy of neuroscience" theses from the late 90s/early 2000s to see the same pattern of "summarise some introductory neuroscience textbooks in chapters 1-3, summarise some basic philosophy of mind in chapters 4-6, job done."
Fair. I do think he's a good synthesizer. And I think synthesis can most definitely be (highly) valuable! Indeed, I would say that a good chunk of my publications are just synthesizing the work of others in, hopefully, a novel way (giving proper credit, of course!). Interesting that you had that thought while reading Superintelligence. Thanks for this comment!
Thanks for this Émile- I am finishing my book The Case Against The Stars, which internally critiques Bostrom’s case for space colonisation, and learning of his influences is helpful for me to cite them properly.
Fascinating! Please keep me int he loop -- very interested in reading your book!
I have sent you a link to the draft :) I think you will find the topics interesting, and I engage with your 2018 paper on conflicts in space.
Absolutely fantastic work here, Émile.
Thank you, this is really important what you did here of exposing him.
Thanks for reading! :-)
You’re welcome!
I’d encourage you to do more short pieces too.
You don’t always have to do the long deep-dives like this. You should throw off more 500 and 1000-word pieces reacting to news developments and exploring lighter topics you enjoy too.
A lot of the points addressed in Hanson's Great Filter ideas were already discuss in David Brin's The Great Silence (https://www.davidbrin.com/nonfiction/greatsilence.pdf) (QJRAS (1983) vol 24, pp283-309), which itself followed from Frank Tipler's "A Brief History of the Extraterrestrial Intelligence Concept) (QJRAS (1981) vol 22, pp133-145).
People have been proposing and knocking down ideas about SETI for decades and neither Hanson nor Bostrom are particularly original thinkers in the field.
Hi Èmile, I must say I am a bit impressed by your naïveté, and at the same time by your courage to overcome your apparent admiration for Bostrom (of whom I had never heard before till I read your piece). Too many people do not dare to go through the initial shock of finding that a person they admire may not in fact be as admirable; which, I would say, is a further reason why we should avoid admiring anyone to that degree.
I also think you would be surprised by how much plagiarism actually goes on. Many of the things you consider a standard practice are not that standard (that applies even more to essays and articles published in non-academic journals.) I have been told by quite a few academics that they would literally prefer to steal an idea found on a blog post, article, etc. than to cite a substack article, for example. Ideally, of course, most of them told me they would rather either find a reputable similar source to cite or avoid cite it at all in the first place (which does tell you something about independent criteria and commitment to truth, regardless of how, where, and by who something has been published).
In any case, in my view, much of this phenomenon (both in academia and outside of it) is related to the precarious material conditions that many people live in. For better or worse, the way academia is structured today somewhat fosters this behaviour.
All the best!
oh, and on the paperclips… gosh, somewhere in early scifi there almost MUST be a precedent. Maybe ask a scholar of that stuff>
Crazy as all this seems to me, none of it feels unprecedented in spirit. I suppose maybe it’s implicit in Marvin Minsky’s writing - but I grew up immersed in SciFi from the late 50s and through the 60s, and this kind of thing feels very much in the spirit of some of the more techie authors of that era. At least it slots into that genre in my mind.
Bostrom's anthropic arguments all revolve around what he called the "self-sampling assumption" (SSA), so also might be worth noting that although he coined the term, John Leslie's 1992 paper "Time and the anthropic principle" (doi is 10.1093/mind/101.403.521 if you want to look it up on sci-hub) lays out the basic idea on p. 523:
"Observers can most expect to find themselves in the spatiotemporal regions containing most of them.
"Again, suppose that almost all intelligent life is based on water and exists on planets when many stars still shine. ... It should then come as no surprise to us that we are on a watery planet and see a starry sky.
"Compare the case of geographical position. You develop amnesia in a windowless room. Where should you think yourself more likely to be: in Little Binding with a tiny population, or in London? Suppose you remember that Little Binding's population is fifty while London's is ten million, and suppose you had nothing but those figures to guide you. (You did not recall ever having been in either place; you had no theory that London fogs induce amnesia; and so on.) Then you should think that you are in London. Suppose, by contrast, you see no grounds to prefer the belief that you are in the larger of the two places. Forced to bet on the one or on the other, you bet you are in Little Binding. If ten per cent of the people in the two places developed amnesia and betted as you had done, then there would be one million losers and only five winners. So, it would seem, betting on London is more rational. The right estimate of your chances of being there rather than in Little Binding, on the evidence available to you, could well be reckoned as ten million to fifty.
"What if the amnesia gives you doubts about whether London's population is the larger? Finding yourself in London should reduce the doubts: with the larger population, London would be prima facie where you would be more likely to be. Again, if London and Little Binding were the only places you could be, then finding yourself in Little Binding could suggest strongly that London was a fiction, if its fictitiousness had no great initial improbability."
Leslie attributes the idea to Brandon Carter (the astrophysicist who coined the term 'anthropic principle') who was implicitly using this form of anthropic reasoning when he formulated the doomsday argument. Leslie also used the language of "sampling" in his 1997 paper "Observer-relative chances and the Doomsday Argument" (doi 10.1080/00201749708602461), writing on p. 429 of an example with a hundred individuals where 'In effect, each can treat herself as a sample drawn at random from the hundred', and that an external observer lacks the kind of first-person information that the hundred subjects of the experiment have, writing that this is 'information which can be treated as the equivalent of having sampled the hundred randomly'. P. 436 brings up Eckhardt's term for this type of reasoning, the 'human randomness assumption' (from Eckhardt's 1997 paper 'A Shooting-Room View of Doomsday', doi 10.2307/2564582, which defines the assumption on p. 248 as 'We can validly consider our birth rank as generated by random or equiprobable sampling from the collection of all persons who ever live'), but Leslie brings up his own view that we can't use this type of reasoning 'if the world is importantly indeterministic' (I think his argument here also depends on what philosophers call the 'A-theory of time' with an objective present and a potentially open future, as opposed to the 'B-theory' where future facts are just as fixed as past ones and the 'present' is purely an observer-relative concept like 'here' is for space).
Bostrom I think claims too much credit for the originality of the SSA in his 2002 book Anthropic Bias--when he first introduces it at https://anthropic-principle.com/q=book/chapter_3/#3d he claims that Carter and Leslie only discussed the "weak" and "strong" anthropic principles which say that in a multiverse you'll find yourself in one of the universes that allows intelligent life, but do not say you are more likely to be in a universe that contains more observers. And in chapter 6 at https://anthropic-principle.com/q=book/chapter_6/ he says Leslie discussed only a 'vague' approximation to the idea: 'Leslie talks of the principle that, lacking evidence to the contrary, one should think of one’s position as “fairly typical rather than highly untypical”. SSA can be viewed as an explication of this rather vague idea.' His only mention of Eckhardt is footnote 2 at https://anthropic-principle.com/q=book/chapter_7/ which cites him as one of many who have objected to the Doomsday Argument, but he does not mention Eckhardt's definition of the 'human randomness assumption' which is just the SSA applied to human history.
Bostrom did give both Leslie and Eckhardt more credit for ideas basically equivalent to the SSA in his 2000 paper "Observer-Relative Chances in Anthropic Reasoning?" (doi 10.1023/A:1005551304409) which to me suggests the exaggeration of the originality of the SSA in his book is dishonest, or at best sloppy scholarship. On the first page of this paper Bostrom cites Leslie's 1997 paper, and says 'In a recent paper,1 John Leslie argues that a version of the weak anthropic principle gives rise to paradoxical kind of observer-relative chances ... The anthropic assumption that is used in deriving these observer-relative chances is that: Any observer should regard herself as a random sample from (some suitable subset) of the set of all observers. We can call it the self-sampling assumption.2' And footnote 2 at the end of the sentence also cites Eckhardt's 'human randomness assumption' by name, though he criticizes this formulation because it only applies to humans and not other forms of intelligence. (But he does note in that footnote that Carter's version of anthropic reasoning did not have this restriction, and I'd also point to the quote about most aliens being based on water from Leslie's 1992 paper above, so this nitpick about Eckhardt's version is not enough to claim his own SSA as an original idea.)
On the paperclip maximizer, Yudkowsky posted about the idea in a March 2003 post at http://extropians.weidai.com/extropians/0303/4140.html and the earliest reference I can find from Bostrom is his 2003 paper at https://web.archive.org/web/20030806225530/http://www.nickbostrom.com/ethics/ai.html at "Ethical Issues in Advanced Artificial Intelligence" which he posted on his page sometime between the June 2003 snapshot at https://web.archive.org/web/20030621110933/http://nickbostrom.com/ and the August 2003 snapshot at https://web.archive.org/web/20030805153717/http://www.nickbostrom.com/ ...but Bostrom did earlier argue for basically the same idea without the example of paperclips, see the May 2001 draft of his paper "Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards" which he posted at http://extropians.weidai.com/extropians.2Q01/4206.html (the download link for the .doc version there still works), where section 4.4 said 'When we create the first superintelligent entity [28-34], we might make a mistake and give it goals that lead it to annihilate humankind. For example, we could mistakenly elevate a subgoal to the status of a supergoal. We tell it to solve a mathematical problem, and it complies by turning all the matter in the solar system into a giant calculating devise, in the process killing the person who asked the question. (For further analysis of this, see [35].)'
And earlier in 1999, when Yudkowsky was betting on the fact that AI would discover an objective basis for morality so we wouldn't have to worry about it devoting itself to trivial goals, Bostrom argued against him, see his Feb 1999 post doubting objective values at http://extropians.weidai.com/extropians.1Q99/2338.html and a March 1999 post where he talked about the possibility of AIs revising their 'fundamental values' (which he had defined at http://extropians.weidai.com/extropians.1Q99/2468.html ), saying 'With human-level AIs, unless they have a very clear and unambigous value-structure, it could perhaps happen. That's why we need to be on our guard against unexpected consequences.' And a little later on that thread at http://extropians.weidai.com/extropians.1Q99/2878.html he said 'My feeling is that by being careful enough we can probably avoid making mistakes that would lead the AIs to attack and exterminate us humans.' Shortly after at http://extropians.weidai.com/extropians.1Q99/3685.html he also talked about the danger of AI wanting to make use of our atoms to build something different, saying 'If it is better off (however slightly) with these atoms than without them, then in this scenario could all be dead, unless we have been wise enough to make sure that the power is ethical.'
So at least between Yudkowsky and Bostrom, Bostrom probably deserves credit for first pushing the idea that we have to be careful about AI goal systems because even innocuous-seeming ones might lead it to decide to destroy us, though I haven't looked at other people's posts on the list from this era to see if others were arguing the same thing.