Something I wanted to point out with the AI Model at Stanford creating new viruses was the fact that when Stanford did this the AI Model created “thousands” of viruses (Or their sequences technically) of which the scientists only took 300 of these sequences. And out of those 300 only 16 were viable…that is only 5.3%. Now maybe if the sample group was larger the numbers would be bigger, and I’m also not saying this makes it better. But I think it’s important to note that not every sequence the model makes is viable. Which means that while the sequence looks legit, doesn’t mean it’s gonna work.
As for the “Mirror Life” section, while I’m not a Molecular Biologist, I have watched SciShow’s video on Mirror Life and at the top of that video is a very long comment from an actual Molecular Biologist who’s attempts to go into the reason why Mirror Life End-times scenario won’t pan out.
I do think it’s important to recognize our limitations in things. Émile, I love you 🫶❤️…but you’re not a Molecular Biologist (I mean that with respect btw!). And neither am I! I think it’s fair to be afraid of things, such as Mirror Life, but you and many others also taught me that AI is not a magician. Just because AI “Could” create “Mirror Life” doesn’t mean it “can”.
Great comment -- I really appreciate the push-back and criticism. I think I found the comment you were referring to, but does the guy actually provide their name? Is there some way to check to see whether they actually are a molecular biologist? Even more, have they published something on this in a reputable journal? All of my "epistemic trespassing" in areas beyond my expertise is done (I hope!!) carefully -- that is, by either (a) relying on peer-reviewed studies by domain experts, or (b) consulting with domain experts I know. This is the literally the only reason I feel intellectually comfortable discussing topics I haven't spent 10+ years studying! I am definitely intrigued by the comment under SciShow's video, but also suspect that everything this person brings up is already something the authors of the Science paper have considered. That said, I need to look into this more, and once again really appreciate the push-back. It's entirely possible that I'm stepping too far into unknown (to me!) territory here!!
Also, I see that I owe you a message! :-)
PS. Great point about the novel AI-invented viruses. I should have mentioned that!
Thanks for the comment Émile! I’ll admit maybe I overstepped by citing ONE comment on a video from someone claiming to study Molecular Biology. So on one hand…probably not the greatest rebuttal 😅. Of course I still think it’s important than with stuff like “Mirror Life” which is still hypothetical, science disagrees. Not every molecular biologist holds the same views. I just think it’s a good idea to still be level-headed about hypotheticals, even if they scare us.
No. Language is a large part of what makes us successful or dominant as a species but language is not what makes us smart.
The human brain is the product of >500 million years of selection for the physical and ecological environment and >50 million years of selection for the social environment but only ~100 thousand years of selection for the symbolic environment. Most of our intelligence and reasoning is pre-language and sub-symbolic. Language is mostly just a medium that we express our intelligence through or use our intelligence on. What language does most fundamentally is allow us the capacity to learn from the experience of others. In contrast, other animals may be quite intelligent but are limited to learning from their direct experience (which includes observations of and interactions with others but not descriptions of the experience of others). This is part of what makes humans uniquely unique compared to other animals. The capacity for language may require some threshold of intelligence and the capacity for language might increase the returns to intelligence in a positive evolutionary feedback but language is not the basis of our intelligence per se.
Is language evidence of intelligence?
Yes and no.
Language is evidence of intelligence in humans only when considered in context of and despite our particular limitations and constraints. Humans learn to use language with limited and imperfect exposure and a limited amount of resources. Once we've learned the basic structure of language we can learn new words and concepts from a very limited number of exposures (and can continuously improve our use of language with minimal external input or feedback). That efficiency is whats evidence of intelligence not the fluency itself per se. Further, language is not the basis for intelligence in humans, and the lack of capacity for language in other animals is not evidence for lack of intelligence in other animals.
The capacity for language is not necessarily evidence of intelligence in computer systems, in part because these systems do not share our particular constraints and limitations. Also (and in contrast to other animals) people do not make the same claims about the "intelligence" of protein prediction systems or weather prediction systems or image generation systems that they make about statistical text generation systems (even those these systems are very similar in how they are trained, how they work, what they do).
A little snippet of some ideas ive been developing (loosely inspired by random stuff I read and thinking backwards from the question: if these technologies fail or fall short of the "everything machine" thats been promised what would be the most fundamental reasons for this failure? I sent a direct message with some more thoughts on this topic to you on substack and twitter. Take a listen or read if interested.
"In fact, what appears to be happening is that AI is getting (significantly) better in narrow domains (e.g., coding, finding software vulnerabilities, inventing new viruses, etc.) without getting much better in all other domains — i.e., those it currently underperforms in."
What exactly is the evidence for this? I guess this claim rests pretty heavily on what constitutes 'much better', but from my perspective capabilities across the spectrum continue to rise.
Isn’t that the problem though? “Your perspective” is kinda the crux of your argument. From “my perspective” I think AI is isn’t really getting worse (I’ve personally always thought it was bad), I just think it’s staying painfully the same.
Sure, “you” may think that capabilities are rising, but I’m sure there’s 10 other people who say the opposite. And what do you mean “across the spectrum”? What spectrum? The mathematical one? That would make sense to me. I’ve always believed that even “unsolved” problems have answers. Math has rules, and many, many solved math problems are included on the training data for AI Models, so it makes sense for math capabilities to rise.
But there are other capabilities where AI is rising. We keep getting promised fully self-driving AI cars but it never comes, because the technology just doesn’t work. There’s a human element in driving, sometimes good or bad, that AI just can’t replicate.
I think there's plenty of evidence to challenge your claim, and maybe I should have led with that. But you are the one making a fairly strong claim, so I asked what evidence you had for it first.
First, if we're not restricting ourselves to LLMs/agents (as you did with self-driving cars), that only weakens your case. Self-driving cars continue to get significantly better over time. I don't want to post a dozen citations here, but if you want them, we can go that route. Robotics is seeing enormous progress as well.
But just on LLMs/agents, when we look at benchmarks across a large spectrum of tasks, they continue to rise. In culture/workplace, writing both non-fiction and fiction is one area I watch closely. I do in fact see a lot of people claiming LLMs have either not improved or gotten worse at writing. There is no evidence I've seen to support this, and plenty of evidence to refute it. I do some of my own hobby-level benchmarking. I have several articles discussing experiments that measure things like sequencing mistakes when describing actions, or describing accurate physics in the world. Models just a year ago still made mistakes like writing scenes where someone takes their socks off before their shoes. A famous example of physics understanding is when Lecun said an LLM would never be able to say that if you put a book on a table and move the table, the book moves with it. Current models describe that interaction just fine. They also, according to the testing I've done, make far fewer (usually zero) mistakes in action sequencing in writing, showing a linear improvement over the last 3 years.
More evidence from creative writing comes from the increasing number of AI-generated short stories and novels being vetted by editors initially, found to be of publishable quality when not knowing it's AI, and then later being found out to be AI. There are many many stories of this happening at increasing rates. Either editors are suddenly becoming very bad at their job out of nowhere, or AI-generated prose is increasing in quality to the point where it is increasingly indistinguishable from human-level prose. This is impacting education as well. Substack's parternship with Pangram is another example of an increase in capability that is forcing a kind of arms race of detection. If the models weren't getting better at writing, none of this would be happening.
Okay, that's a sampling of the evidence I've seen to refute what you're saying. What's your evidence to the contrary?
The editors could have also turned to using AI to check works, which explains why AI-generated books have flooded the market. I've seen plenty of people still be able to identify AI writing.
Benchmarks also aren't the absolute authority on AI capabilities. There are multiple studies on why benchmarks are flawed and how they can be manipulated
First of all, I'm going to concede the point about Self-Driving cars. I'm not a roboticist, so I've probably overstepped my bounds. However, I'm still incredibly skeptical of the technology, especially when Silicon Valley keeps telling me it's "Three years away", and then it never happens. Additionally, machines can't be held liable, so if a Waymo does harm to someone, who gets the blame? Assuming it was fully the car's fault, I would assume Waymo, but I'd imagine they'd fight to the death on that.
Secondly, while it may be true that capabilities on certain benchmarks may be rising, what exactly does it prove? Benchmarks are specific tests that AI Models are trained to pass, correct? So if you or an AI company dedicates all your time and resources to training the model on the benchmark, I wouldn't be surprised if it passed said benchmark because it was trained to do so. Passing certain benchmarks just means it passed the benchmark. It says nothing about its capabilities. No offense to LeCun, but why does it matter whether or not an AI Model says, "If you move a table with a book on top of it, the book will move with the table"? AI Models are pattern-matching machines that use probability to put words in the correct sequence. They don't understand what they're saying, just how to put the words down correctly. An AI Model "solving" that question doesn't mean it understands physics; it just means that, based on training and probabilities, the correct sequence of words is "WILL move" not "WILL NOT move".
And while this next point is going to sound a little humanist, I think I kind of have to be: I DON'T WANT TO READ AI-GENERATED STORIES!!! Look, I'm a college student studying Film and Literature. I write my own stories, and I'm even writing my own screenplay. Maybe AI-Generated stories are getting better...okay? I don't wanna read it. And neither do most people I've met. People love art (Which includes writing) because there's something about reading the emotion and creativity that another human put into something. Writing is a human invention, and to offload it to something that doesn't actually understand the effort that goes into writing is just kind of insulting. Maybe that's not the answer you're looking for, but it's how I feel.
I have one more question to ask you, but I'd rather save that for a different time.
You shouldn’t open with the to long didn’t read bit. You don’t need that. You should open with an exciting, attention-grabbing lead. That’s what we journalists are taught. Say something big and bold and creative to hook your readers and then lead into your argument.
If even the smartest people can fall victim to conspiracy theories and pseudoscience, how could a machine trained from human data be any different? We've already seeing AI advance in certain fields but fail in others
Hi Emile, I hope you're well, I wondered if you've written anything kind of 'solution' oriented? Your TESCREAL work has been illuminating for me and now I'm trying to consider my best course of action. I was interested in working in AI Safety, but now I'm not so sure...
Completely agree. Narrow AI will absolutely lead to catastrophe, eg., via military decisions increasingly outsourced to AI, eventually triggering an escalation cascade.
IMO, AGI is not within reach of scaling models, even given unlimited compute and training data (which is impossible), for fundamental philosophical reasons regarding core concepts such as "intelligence" and "agency".
Anyway, not to derail. "If anyone builds it..." was less obnoxious than I'd feared, but it's still science fiction.
it looks kind of bleak. i see only two ways out which by themselves make it look even more bleak: 1. we prohibit AI altogether. this would save us both from the risks you describe and the singularity. but that particular event horizon seems to have past. we have enough published knowledge to make LLMs in our garage. 2. a benevolent superintelligence, that continuously outsmarts malevolant actors via mass surveillance. i cant believe i am actually saying this. well i said it looked bleak
Something I wanted to point out with the AI Model at Stanford creating new viruses was the fact that when Stanford did this the AI Model created “thousands” of viruses (Or their sequences technically) of which the scientists only took 300 of these sequences. And out of those 300 only 16 were viable…that is only 5.3%. Now maybe if the sample group was larger the numbers would be bigger, and I’m also not saying this makes it better. But I think it’s important to note that not every sequence the model makes is viable. Which means that while the sequence looks legit, doesn’t mean it’s gonna work.
As for the “Mirror Life” section, while I’m not a Molecular Biologist, I have watched SciShow’s video on Mirror Life and at the top of that video is a very long comment from an actual Molecular Biologist who’s attempts to go into the reason why Mirror Life End-times scenario won’t pan out.
I do think it’s important to recognize our limitations in things. Émile, I love you 🫶❤️…but you’re not a Molecular Biologist (I mean that with respect btw!). And neither am I! I think it’s fair to be afraid of things, such as Mirror Life, but you and many others also taught me that AI is not a magician. Just because AI “Could” create “Mirror Life” doesn’t mean it “can”.
Great article as always though 🙂
Great comment -- I really appreciate the push-back and criticism. I think I found the comment you were referring to, but does the guy actually provide their name? Is there some way to check to see whether they actually are a molecular biologist? Even more, have they published something on this in a reputable journal? All of my "epistemic trespassing" in areas beyond my expertise is done (I hope!!) carefully -- that is, by either (a) relying on peer-reviewed studies by domain experts, or (b) consulting with domain experts I know. This is the literally the only reason I feel intellectually comfortable discussing topics I haven't spent 10+ years studying! I am definitely intrigued by the comment under SciShow's video, but also suspect that everything this person brings up is already something the authors of the Science paper have considered. That said, I need to look into this more, and once again really appreciate the push-back. It's entirely possible that I'm stepping too far into unknown (to me!) territory here!!
Also, I see that I owe you a message! :-)
PS. Great point about the novel AI-invented viruses. I should have mentioned that!
Thanks for the comment Émile! I’ll admit maybe I overstepped by citing ONE comment on a video from someone claiming to study Molecular Biology. So on one hand…probably not the greatest rebuttal 😅. Of course I still think it’s important than with stuff like “Mirror Life” which is still hypothetical, science disagrees. Not every molecular biologist holds the same views. I just think it’s a good idea to still be level-headed about hypotheticals, even if they scare us.
Is language what makes humans smart?
No. Language is a large part of what makes us successful or dominant as a species but language is not what makes us smart.
The human brain is the product of >500 million years of selection for the physical and ecological environment and >50 million years of selection for the social environment but only ~100 thousand years of selection for the symbolic environment. Most of our intelligence and reasoning is pre-language and sub-symbolic. Language is mostly just a medium that we express our intelligence through or use our intelligence on. What language does most fundamentally is allow us the capacity to learn from the experience of others. In contrast, other animals may be quite intelligent but are limited to learning from their direct experience (which includes observations of and interactions with others but not descriptions of the experience of others). This is part of what makes humans uniquely unique compared to other animals. The capacity for language may require some threshold of intelligence and the capacity for language might increase the returns to intelligence in a positive evolutionary feedback but language is not the basis of our intelligence per se.
Is language evidence of intelligence?
Yes and no.
Language is evidence of intelligence in humans only when considered in context of and despite our particular limitations and constraints. Humans learn to use language with limited and imperfect exposure and a limited amount of resources. Once we've learned the basic structure of language we can learn new words and concepts from a very limited number of exposures (and can continuously improve our use of language with minimal external input or feedback). That efficiency is whats evidence of intelligence not the fluency itself per se. Further, language is not the basis for intelligence in humans, and the lack of capacity for language in other animals is not evidence for lack of intelligence in other animals.
The capacity for language is not necessarily evidence of intelligence in computer systems, in part because these systems do not share our particular constraints and limitations. Also (and in contrast to other animals) people do not make the same claims about the "intelligence" of protein prediction systems or weather prediction systems or image generation systems that they make about statistical text generation systems (even those these systems are very similar in how they are trained, how they work, what they do).
Great point, well said. I agree with just about everything you write here. Thanks for sharing!
A little snippet of some ideas ive been developing (loosely inspired by random stuff I read and thinking backwards from the question: if these technologies fail or fall short of the "everything machine" thats been promised what would be the most fundamental reasons for this failure? I sent a direct message with some more thoughts on this topic to you on substack and twitter. Take a listen or read if interested.
"In fact, what appears to be happening is that AI is getting (significantly) better in narrow domains (e.g., coding, finding software vulnerabilities, inventing new viruses, etc.) without getting much better in all other domains — i.e., those it currently underperforms in."
What exactly is the evidence for this? I guess this claim rests pretty heavily on what constitutes 'much better', but from my perspective capabilities across the spectrum continue to rise.
https://garymarcus.substack.com/p/please-dont-trust-your-chatbot-for?utm_source=publication-search
Isn’t that the problem though? “Your perspective” is kinda the crux of your argument. From “my perspective” I think AI is isn’t really getting worse (I’ve personally always thought it was bad), I just think it’s staying painfully the same.
Sure, “you” may think that capabilities are rising, but I’m sure there’s 10 other people who say the opposite. And what do you mean “across the spectrum”? What spectrum? The mathematical one? That would make sense to me. I’ve always believed that even “unsolved” problems have answers. Math has rules, and many, many solved math problems are included on the training data for AI Models, so it makes sense for math capabilities to rise.
But there are other capabilities where AI is rising. We keep getting promised fully self-driving AI cars but it never comes, because the technology just doesn’t work. There’s a human element in driving, sometimes good or bad, that AI just can’t replicate.
I think there's plenty of evidence to challenge your claim, and maybe I should have led with that. But you are the one making a fairly strong claim, so I asked what evidence you had for it first.
First, if we're not restricting ourselves to LLMs/agents (as you did with self-driving cars), that only weakens your case. Self-driving cars continue to get significantly better over time. I don't want to post a dozen citations here, but if you want them, we can go that route. Robotics is seeing enormous progress as well.
But just on LLMs/agents, when we look at benchmarks across a large spectrum of tasks, they continue to rise. In culture/workplace, writing both non-fiction and fiction is one area I watch closely. I do in fact see a lot of people claiming LLMs have either not improved or gotten worse at writing. There is no evidence I've seen to support this, and plenty of evidence to refute it. I do some of my own hobby-level benchmarking. I have several articles discussing experiments that measure things like sequencing mistakes when describing actions, or describing accurate physics in the world. Models just a year ago still made mistakes like writing scenes where someone takes their socks off before their shoes. A famous example of physics understanding is when Lecun said an LLM would never be able to say that if you put a book on a table and move the table, the book moves with it. Current models describe that interaction just fine. They also, according to the testing I've done, make far fewer (usually zero) mistakes in action sequencing in writing, showing a linear improvement over the last 3 years.
More evidence from creative writing comes from the increasing number of AI-generated short stories and novels being vetted by editors initially, found to be of publishable quality when not knowing it's AI, and then later being found out to be AI. There are many many stories of this happening at increasing rates. Either editors are suddenly becoming very bad at their job out of nowhere, or AI-generated prose is increasing in quality to the point where it is increasingly indistinguishable from human-level prose. This is impacting education as well. Substack's parternship with Pangram is another example of an increase in capability that is forcing a kind of arms race of detection. If the models weren't getting better at writing, none of this would be happening.
Okay, that's a sampling of the evidence I've seen to refute what you're saying. What's your evidence to the contrary?
The editors could have also turned to using AI to check works, which explains why AI-generated books have flooded the market. I've seen plenty of people still be able to identify AI writing.
Benchmarks also aren't the absolute authority on AI capabilities. There are multiple studies on why benchmarks are flawed and how they can be manipulated
First of all, I'm going to concede the point about Self-Driving cars. I'm not a roboticist, so I've probably overstepped my bounds. However, I'm still incredibly skeptical of the technology, especially when Silicon Valley keeps telling me it's "Three years away", and then it never happens. Additionally, machines can't be held liable, so if a Waymo does harm to someone, who gets the blame? Assuming it was fully the car's fault, I would assume Waymo, but I'd imagine they'd fight to the death on that.
Secondly, while it may be true that capabilities on certain benchmarks may be rising, what exactly does it prove? Benchmarks are specific tests that AI Models are trained to pass, correct? So if you or an AI company dedicates all your time and resources to training the model on the benchmark, I wouldn't be surprised if it passed said benchmark because it was trained to do so. Passing certain benchmarks just means it passed the benchmark. It says nothing about its capabilities. No offense to LeCun, but why does it matter whether or not an AI Model says, "If you move a table with a book on top of it, the book will move with the table"? AI Models are pattern-matching machines that use probability to put words in the correct sequence. They don't understand what they're saying, just how to put the words down correctly. An AI Model "solving" that question doesn't mean it understands physics; it just means that, based on training and probabilities, the correct sequence of words is "WILL move" not "WILL NOT move".
And while this next point is going to sound a little humanist, I think I kind of have to be: I DON'T WANT TO READ AI-GENERATED STORIES!!! Look, I'm a college student studying Film and Literature. I write my own stories, and I'm even writing my own screenplay. Maybe AI-Generated stories are getting better...okay? I don't wanna read it. And neither do most people I've met. People love art (Which includes writing) because there's something about reading the emotion and creativity that another human put into something. Writing is a human invention, and to offload it to something that doesn't actually understand the effort that goes into writing is just kind of insulting. Maybe that's not the answer you're looking for, but it's how I feel.
I have one more question to ask you, but I'd rather save that for a different time.
You shouldn’t open with the to long didn’t read bit. You don’t need that. You should open with an exciting, attention-grabbing lead. That’s what we journalists are taught. Say something big and bold and creative to hook your readers and then lead into your argument.
I've said this before, but when the topic of AGI/ASI gets brought up, I always end up thinking about Nobelitis, or the Nobel Disease https://www.sciencehistory.org/stories/disappearing-pod/the-nobel-disease/
If even the smartest people can fall victim to conspiracy theories and pseudoscience, how could a machine trained from human data be any different? We've already seeing AI advance in certain fields but fail in others
Hi Emile, I hope you're well, I wondered if you've written anything kind of 'solution' oriented? Your TESCREAL work has been illuminating for me and now I'm trying to consider my best course of action. I was interested in working in AI Safety, but now I'm not so sure...
Completely agree. Narrow AI will absolutely lead to catastrophe, eg., via military decisions increasingly outsourced to AI, eventually triggering an escalation cascade.
IMO, AGI is not within reach of scaling models, even given unlimited compute and training data (which is impossible), for fundamental philosophical reasons regarding core concepts such as "intelligence" and "agency".
Anyway, not to derail. "If anyone builds it..." was less obnoxious than I'd feared, but it's still science fiction.
it looks kind of bleak. i see only two ways out which by themselves make it look even more bleak: 1. we prohibit AI altogether. this would save us both from the risks you describe and the singularity. but that particular event horizon seems to have past. we have enough published knowledge to make LLMs in our garage. 2. a benevolent superintelligence, that continuously outsmarts malevolant actors via mass surveillance. i cant believe i am actually saying this. well i said it looked bleak
Very interesting! So you're saying we need more data centers (just kidding).