Biosecurity expert here, I think your article is as wrong as Noah’s.
Yes, computational design is hard. No, the transmissibility/lethality tradeoff is not guaranteed, especially not for cleverly made viruses. Virologists know how to do it, they just (thankfully) don’t include it in the training data. Great example is smallpox and a few other pathogens wiping out nearly 95% of native Americans.
There’s a reason the BWC was easy to negotiate in an era of nuclear proliferation: any country that has gone down this road has realized it’s the path to dooming humanity.
Bio is way worse than nukes for those of us who really know bio from a security perspective. LLMs might not be the enablers (thankfully they’re not trained on classified corpuses from bioweapons facilities of old, such as Russia’s smallpox work or some other agents I don’t even want to mention).
Noah’s article is wrong because it overestimates AI. Yours is wrong because it underestimates bio.
‘Virologists know how to do it, they just don’t include it in the training data’ seems a pretty fantastical claim. Interested in hearing more about this.
Thanks for the response. I don’t doubt that the theoretical upper ceiling of disease/virus lethality is unimaginably destructive, or that biowarfare is a terrifying thing. That said, I hope you can understand why any claim about a group of scientists having secret knowledge of something devastating is generally going to be something that invites skepticism, hence my original comment.
I’m also not going to pretend I’m an expert on the transmission/virulence tradeoff, but I am also wary of arguments appealing to semantics about hypotheses vs laws and such. Biology is sufficiently complex such that there’s very little in our field that qualifies as an absolute physical law (all models are wrong but some are useful, etc). Even looking at Mendel’s laws of inheritance, for instance - does the existence of something like incomplete dominance mean we can handwave Mendel’s initial framework, because of the assumptions he made? Either way, I won’t speak further to this point because once again, my background is not in virology and I won’t claim otherwise. Just not swayed by the argument about it being ‘just a hypothesis’ I’m afraid.
I’ll end here by saying I get your position and why you feel you’re not at liberty to say more, but it’s still hard for me to believe the claim that this article is *just as wrong* as one claiming LLM-driven virus research is what’s going to kill us all. At the cell level, something like COVID has had untold trillions of brute-force chances to mutate into something even more evolutionarily successful since 2020 (and it even did so at a few points!) and I’m not yet convinced someone with an LLM is going to outpace that. Maybe I’ll be wrong. Who knows.
Thank you for sharing your perspective. My point was that both articles are wrong in their own way. Noah’s article overestimates AI, this article underestimates bio.
Can AI pave the path for enabling people to do bad things in bio? Sure. Is it necessary? No. Bad-enough things could be done in the 1960s before we even knew DNA was the object of inheritance that even Richard Nixon got freaked out, said bio is horrible, and sought signatories of the BWC. Thankfully, old school methods aren’t widely known and are overlooked in the hype around AI. Thankfully we didn’t have the internet back then and the literature from then isn’t online.
I agree with your skepticism of biological laws. The people who believe bio can’t kill us all rely heavily on the myxomatosis example and a few mathematical models derived from that case study. Plus, some look at studies of influenza passage and tissue tropism, as I mention. Those n=2 examples and models made around them are the scientific foundation on which some say bio can’t kill is all. It misses many other examples even in nature where bio did kill most things, where transmission dynamics and evolutionary pressures differ consequentially from those constrained by influenza’s sialic acid binding, and what someone could do or what people did do in the 1950s and 60s leading governments to agree, amidst nuclear weapons proliferation, to stop developing biological weapons.
Smallpox is just one pathogen of many that isn’t in the myxomatosis/influenza models. Why was smallpox so virulent? How did European diseases kill so many native Americans? Questions worth mulling over
I'm curious about the statement: "the most cunning biologists know how to do it."
Is the limiting factor actually "cunning"? I.e. with currently public knowledge, a sufficiently intelligent biologist would be about to figure out how to design a bioweapon? Or is it actually knowledge, which requires experiments in secret facilities like the ones in the former Soviet Union to acquire?
If the former, I might be worried, since the rate of growth of LLM intelligence continues to be impressive, so even if Noah is wrong now, he might be right in a few years. But if it requires gathering more data in the physical world, then I would be inclined to be less worried. (Of course, LLMs are pretty good at hacking and are getting better, so I hope the classified data is well secured.)
Current technologies are sufficient. Most people don’t know what was done before the BWC, but could reinvent the wheel quite easily with enough subject matter expertise. So yes, LLMs could potentially enable someone to do dangerous things, but it won’t require synthetic imputation of a novel virus, just clever use of old tech.
Well, the BWC was easy to negotiate, but the Soviets clearly didn’t take part during the Cold War. As I mentioned in my comment on this thread, I’m reading Annie Jacobsen’s new book on biological warfare, and she reports that the current US assessment is that Russia has continued the Soviet bioweapons program up to the present. The scenario she presents involves genetically-modified pneumonic plague causing hundreds of millions of deaths and causing civilizational collapse worldwide. I’d be curious to know if you find this plausible.
Noah's take was an instant unsubscribe for me. I don't mind when smart people poke around at the edges of their expertise - Lord knows I do it. But that felt wildly arrogant and showcased a lack of humility that I find untrustworthy.
I largely disagree with Noah Smith and mostly agree with you. I also think Smith's blog's title was irresponsible.
I don't worry much about civilization-ending viruses being synthesized in a lab. But, I do worry about AI making it easier to design and create SARS-CoV-2-level viruses. If it does and pandemic-potential viruses are being unleashed every few years, that would be awful. I think we all remember how miserable 2020 was WRT loss of life and disruption to society and the economy.
The evidence makes it clear that SARS-CoV-2 was not designed in or leaked from a lab. However, I think that the lab-leak hypothesis endures in part because it probably could have been. SARS-CoV-2 was just some bat/pangolin virus that needed a handful of unlucky changes to cause a human pandemic. Bad actors with AI might be able to catalyze these kinds of changes.
I'm less worried because I think now and for the foreseeable future the "handful of unlucky changes" are extremely difficult to rationally predict, AI or not. We just don't know how to tune a virus so it becomes more or less lethal, or more or less transmissible, and how to get it just right.
Just out of curiosity (also an economist, so definitely no expert here!) - why wouldn't it be a problem that there is more compute capacity, making it possible to try out many more variations? Like, the AI doesn't need to have an understanding of what's going on / perfectly be able to predict which lever will be the right one to pull. If it can just go through many more cases much faster, it might be able to find the right covid like virus?
(That being said i very much agree with you, that people will respond. So civilizational wipe-out is unrealistic)
What exactly do you envision when you say "try out many more variations"? AI cannot predict whether a variation is viable or not, so it cannot actually perform this search for the right virus.
Right, I think I naively overlooked that virus design eventually needs to be taken to the physical world, where the properties need to be verified and tested... unlike say maths where for a certain proof you "just" need to find a contradicting example, or cybersecurity where to "hack" a password you don't necessarily need to be smart at decryption if you could try a million different options by brute force
Excellent piece. I don't know who Noah is or what he does, but I read the first paragraph of his "story" and I stopped. As someone who has been trained in computational system biology, who has worked in AI/ML for the past 20 years, the fear mongering around AI really annoy me.
Fight, fight against Gell-Mann amnesia! You can do it!
I have Noah Smith grouped with Richard Hanania in my mind: people who occasionally write very smart things about subjects they know very well, and routinely write rather silly pieces about subjects they know much less about. I subscribe to both on substack, but I rarely end up reading more than the first few paragraphs of their essays.
I agree with all your takes. I do think that COVID, but with the inverse age-skew (towards children, not elderly), could have given civilization a real run for its money. Purely from the social fallout
A big part of the problem is a lot of people had their brains melted during COVID and became convinced that SARS-CoV-2 was a deliberately engineered bioweapon, so I guess it’s not much of a leap to “AI would make that even easier.” Also, a lot of these AI boosters/enthusiasts think they understand every field outside their own, even better than subject matter experts, and sycophantic LLMs reinforce that notion.
I tend to cringe when a scientist says it is impossible to "practically" make or do such and such. Here I don't include those things that are physically impossible (like, accelerating to the velocity of light, or perpetual machines with no energy input). Time and time again human ingenuity had falsified proclamations of limits to practical attainment.
There is no a priori reason to assume there can be no human virus with 90% lethality. True, there is no such *known* human virus: it could be unknown because such a virus might be rather unfit to propagate through to a pandemic status or because of human genetic diversity. But there are bird flu types (H5N1, H5N8) that are 90-100% lethal to (admittedly genetically nearly homogeneous) poultry populations. There is zero assurance that a virus cannot be engineered, which uses a highly invariant human cell surface protein as a receptor, such as the Na-K ATPase transporter subunit or the primary glucose transporter. Whether a high-school student can make it? I don't know but I never underestimate the younger generations.
Not just 90% lethality. 90% lethality while spreading easily and spreading primarily during the asymptomatic period, so nobody notices they are infected until it's too late. I think such a scenario is extremely unlikely. Maybe not impossible, but sufficiently unlikely that I'd rather worry about other things.
And, in any case, the only thing we can do to protect ourselves from such a virus is to improve public health and invest in rapid vaccine development.
Feels like the biggest threat from AI in relation to viruses is the deleterious effects it can have on public trust and critical thinking.
If we produce a big generation of people with stunted development, incapable of independent thought and lacking the ability to recognise when they’re being exploited, public health campaigns will falter, vaccine rates will plummet, and viruses will have a lot more hosts to evolve in.
Who needs smart programs when you have a dumb populace?
One scenario (which I find implausible) that Yudkowsky describes for AI caused human extinction is the AI creates self-replicating nanobots which get inside and kill everyone on cue. Now, my impression is that we are all under attack by self-replicating nanobots all the time and have been for however many hundreds of millions of years. What are your thoughts on such a scenario?
The best way to evaluate Yudkowsky’s statements is to think of him as a cult leader rather than a scientist. He doesn’t care about plausibility or physical constraints. He only cares about the emotional responses he can evoke.
To add, Yudkovsky never completed any formal education nor ever held a job. Which means he never had a person of both greater competence and authority to tell him: your work is crap, go redo it or you get an F / lose your job.
He's basically an ignorant grifter with a gift of gab.
Setting aside the issue of AI-engineered pathogens for a second, I'm currently reading Annie Jacobsen's "Biological War: A Scenario", and I'm wondering how plausible you find that. The Russians apparently have a biological warfare program carried over from the Soviet era, which has weaponized pathogens like smallpox, Marburg, and plague. In her fictional scenario, a Russian scientist feeling guilty tries to destroy the his research and inadvertently ends up freeing pneumonic plague which has been genetically modified to induce euphoria in the early stages to keep the patient active and spreading even when they begin to get symptomatic. This is supposed to be well-researched and based on factual information from defecting Russian scientists. In the scenario, this basically leads to complete society breakdown and mass death. Is this plausible to you?
I’m still curious to hear your response to Soviet/Russian engineered pathogens and how bad you think they might be. Your article seems to portray such scenarios as highly implausible.
Bioweapons research is typically not about making pathogens worse (they are bad enough already), it's about delivery systems and control. Can you somehow infect an entire city at once? Can you infect the enemy while you protect ourself? A smallpox release would be pretty bad though, in particular now that most people are no longer vaccinated. No engineering required.
My main point remains: The stuff that exists naturally is as bad as or worse than anything engineered. Let's worry about naturally occurring pathogens first.
Yeah, I feel like you didn't really read my first comment or address my question. They genetically modified pneumonic plague to make the infected person euphoric during early onset. This addresses exactly what you're talking about. The euphoria makes the infected person continue to be active and social even through the onset of symptoms, increasing spread. They also did gain-of-function modifications to increase the R0.
Again: Do you find this plausible? Both the feasibility of the modifications and that they would wreak far more damage upon release?
Why would the Soviets/Russians go to so much effort to genetically modify these pathogens if naturally occurring ones are as bad or worse. I'm not a virologist, but I can't see how it is possibly the case, and as far as I can tell you have not given any kind of reason why this would be.
Let's start with the fact that pneumonic plague is not caused by a virus.
I don't know anything about this specific research so I can't comment. In general I'm wary of third-party recollections of research because often what is being described is quite different from what the actual research was.
Masks work great against bacteria and so I expect if there was an outbreak of pneumonic plague people would start masking and that would bring the outbreak under control rather quickly. Also, antibiotics work against bacteria, unless you now also throw in genetic engineering for drug resistance.
Would some people die in such an outbreak? Yes. Would it be a world-wide disaster? Probably not.
Yes, but according to Dario Amodei it “will actually be possible to cure most human disease in ~5-10 years” Given the biological knowledge gain implied by this claim, maybe Noah is onto something. Maybe if this claim and others like it turn out to be wrong, then we’ll still be discussing hypotheticals 10 years from now. I expect to be reading hypotheticals (and AI post-mortems), 10 years from now.
If the definition of supervirus is “kills 90% of the human population” (or even just 10%) then no, measles won’t do that. A world-wide measles outbreak would be bad, but it wouldn’t end civilization.
Am I wrong for understanding the pattern of lethality and transmissibility in nature is a consequence of the evolution rather than an inherent trade-off in the virus design?
I can believe there is an inherent trade off in long asymptomaticity and transmissibility as you justified.
None of this really comforts my worry that something resembling a more dangerous version of COVID could be made.
Arguing that in nature, viruses are rarely highly lethal and transmissible means it will be difficult to engineer is analagous to arguing against wheeled locomotion on the grounds that it doesn't occur as a transport method in nature. I think the point is that there are other evolutionary constraints that push nature to be like nature and nature is rarely a good upper bound on "the fastest way to fly", "the most energy efficient way to move" or "the deadliest possible virus".
Am I wrong for believing that this is conceivable: some (much smarter) future AI system use advanced computational modelling, improve the algorithms themselves, to generate 1000 promising virus candidates that are tweaked, more lethal versions of COVID?
The SARS genome is only 30K base pairs (60K bits of information); this is a much smaller search space than say, an image (obviously you know more than I do about the relative difficulty of searching this space but the remaining obstacles really seem down to ample training data and modelling of changes).
Assuming this is biologically possible: could the shear number of viruses prevent effective response, would we be able to stop a misaligned AI from doing this?
I take your point that computational modelling isn't at this stage yet; when, in your opinion, is the correct time to panic, my leading candidate is this: if AI starts making meaningful progress on computational biology, especially if some kind of rapid testing and feedback system is established on physical beings.
But your argument (however correct it may ultimately be) starts with "I'm a computational biologist and used to evaluate AI systems, and let me tell you MY FIELD IS SO SO SO SO HARD, MAN IT IS SO GODDAMN HARD!!", and this is just so funny in 2026.
Biosecurity expert here, I think your article is as wrong as Noah’s.
Yes, computational design is hard. No, the transmissibility/lethality tradeoff is not guaranteed, especially not for cleverly made viruses. Virologists know how to do it, they just (thankfully) don’t include it in the training data. Great example is smallpox and a few other pathogens wiping out nearly 95% of native Americans.
There’s a reason the BWC was easy to negotiate in an era of nuclear proliferation: any country that has gone down this road has realized it’s the path to dooming humanity.
Bio is way worse than nukes for those of us who really know bio from a security perspective. LLMs might not be the enablers (thankfully they’re not trained on classified corpuses from bioweapons facilities of old, such as Russia’s smallpox work or some other agents I don’t even want to mention).
Noah’s article is wrong because it overestimates AI. Yours is wrong because it underestimates bio.
‘Virologists know how to do it, they just don’t include it in the training data’ seems a pretty fantastical claim. Interested in hearing more about this.
Thanks for the response. I don’t doubt that the theoretical upper ceiling of disease/virus lethality is unimaginably destructive, or that biowarfare is a terrifying thing. That said, I hope you can understand why any claim about a group of scientists having secret knowledge of something devastating is generally going to be something that invites skepticism, hence my original comment.
I’m also not going to pretend I’m an expert on the transmission/virulence tradeoff, but I am also wary of arguments appealing to semantics about hypotheses vs laws and such. Biology is sufficiently complex such that there’s very little in our field that qualifies as an absolute physical law (all models are wrong but some are useful, etc). Even looking at Mendel’s laws of inheritance, for instance - does the existence of something like incomplete dominance mean we can handwave Mendel’s initial framework, because of the assumptions he made? Either way, I won’t speak further to this point because once again, my background is not in virology and I won’t claim otherwise. Just not swayed by the argument about it being ‘just a hypothesis’ I’m afraid.
I’ll end here by saying I get your position and why you feel you’re not at liberty to say more, but it’s still hard for me to believe the claim that this article is *just as wrong* as one claiming LLM-driven virus research is what’s going to kill us all. At the cell level, something like COVID has had untold trillions of brute-force chances to mutate into something even more evolutionarily successful since 2020 (and it even did so at a few points!) and I’m not yet convinced someone with an LLM is going to outpace that. Maybe I’ll be wrong. Who knows.
Thank you for sharing your perspective. My point was that both articles are wrong in their own way. Noah’s article overestimates AI, this article underestimates bio.
Can AI pave the path for enabling people to do bad things in bio? Sure. Is it necessary? No. Bad-enough things could be done in the 1960s before we even knew DNA was the object of inheritance that even Richard Nixon got freaked out, said bio is horrible, and sought signatories of the BWC. Thankfully, old school methods aren’t widely known and are overlooked in the hype around AI. Thankfully we didn’t have the internet back then and the literature from then isn’t online.
I agree with your skepticism of biological laws. The people who believe bio can’t kill us all rely heavily on the myxomatosis example and a few mathematical models derived from that case study. Plus, some look at studies of influenza passage and tissue tropism, as I mention. Those n=2 examples and models made around them are the scientific foundation on which some say bio can’t kill is all. It misses many other examples even in nature where bio did kill most things, where transmission dynamics and evolutionary pressures differ consequentially from those constrained by influenza’s sialic acid binding, and what someone could do or what people did do in the 1950s and 60s leading governments to agree, amidst nuclear weapons proliferation, to stop developing biological weapons.
Smallpox is just one pathogen of many that isn’t in the myxomatosis/influenza models. Why was smallpox so virulent? How did European diseases kill so many native Americans? Questions worth mulling over
I'm curious about the statement: "the most cunning biologists know how to do it."
Is the limiting factor actually "cunning"? I.e. with currently public knowledge, a sufficiently intelligent biologist would be about to figure out how to design a bioweapon? Or is it actually knowledge, which requires experiments in secret facilities like the ones in the former Soviet Union to acquire?
If the former, I might be worried, since the rate of growth of LLM intelligence continues to be impressive, so even if Noah is wrong now, he might be right in a few years. But if it requires gathering more data in the physical world, then I would be inclined to be less worried. (Of course, LLMs are pretty good at hacking and are getting better, so I hope the classified data is well secured.)
Current technologies are sufficient. Most people don’t know what was done before the BWC, but could reinvent the wheel quite easily with enough subject matter expertise. So yes, LLMs could potentially enable someone to do dangerous things, but it won’t require synthetic imputation of a novel virus, just clever use of old tech.
A biosecurity expert he says. And apparently deterrence? This has made my day, thanks Alex.
Hi Saskia, I hope you’re well. Everyone has something to teach 😊
Well, the BWC was easy to negotiate, but the Soviets clearly didn’t take part during the Cold War. As I mentioned in my comment on this thread, I’m reading Annie Jacobsen’s new book on biological warfare, and she reports that the current US assessment is that Russia has continued the Soviet bioweapons program up to the present. The scenario she presents involves genetically-modified pneumonic plague causing hundreds of millions of deaths and causing civilizational collapse worldwide. I’d be curious to know if you find this plausible.
Thank you. A bit of realism.
Noah's take was an instant unsubscribe for me. I don't mind when smart people poke around at the edges of their expertise - Lord knows I do it. But that felt wildly arrogant and showcased a lack of humility that I find untrustworthy.
I think Noah’s difficulty is, like most writers, he cannot understand that some things are infinitely more complex than language.
I largely disagree with Noah Smith and mostly agree with you. I also think Smith's blog's title was irresponsible.
I don't worry much about civilization-ending viruses being synthesized in a lab. But, I do worry about AI making it easier to design and create SARS-CoV-2-level viruses. If it does and pandemic-potential viruses are being unleashed every few years, that would be awful. I think we all remember how miserable 2020 was WRT loss of life and disruption to society and the economy.
The evidence makes it clear that SARS-CoV-2 was not designed in or leaked from a lab. However, I think that the lab-leak hypothesis endures in part because it probably could have been. SARS-CoV-2 was just some bat/pangolin virus that needed a handful of unlucky changes to cause a human pandemic. Bad actors with AI might be able to catalyze these kinds of changes.
I'm less worried because I think now and for the foreseeable future the "handful of unlucky changes" are extremely difficult to rationally predict, AI or not. We just don't know how to tune a virus so it becomes more or less lethal, or more or less transmissible, and how to get it just right.
Just out of curiosity (also an economist, so definitely no expert here!) - why wouldn't it be a problem that there is more compute capacity, making it possible to try out many more variations? Like, the AI doesn't need to have an understanding of what's going on / perfectly be able to predict which lever will be the right one to pull. If it can just go through many more cases much faster, it might be able to find the right covid like virus?
(That being said i very much agree with you, that people will respond. So civilizational wipe-out is unrealistic)
What exactly do you envision when you say "try out many more variations"? AI cannot predict whether a variation is viable or not, so it cannot actually perform this search for the right virus.
Right, I think I naively overlooked that virus design eventually needs to be taken to the physical world, where the properties need to be verified and tested... unlike say maths where for a certain proof you "just" need to find a contradicting example, or cybersecurity where to "hack" a password you don't necessarily need to be smart at decryption if you could try a million different options by brute force
Excellent piece. I don't know who Noah is or what he does, but I read the first paragraph of his "story" and I stopped. As someone who has been trained in computational system biology, who has worked in AI/ML for the past 20 years, the fear mongering around AI really annoy me.
This is an excellent piece, thanks very much.
Fight, fight against Gell-Mann amnesia! You can do it!
I have Noah Smith grouped with Richard Hanania in my mind: people who occasionally write very smart things about subjects they know very well, and routinely write rather silly pieces about subjects they know much less about. I subscribe to both on substack, but I rarely end up reading more than the first few paragraphs of their essays.
I agree with all your takes. I do think that COVID, but with the inverse age-skew (towards children, not elderly), could have given civilization a real run for its money. Purely from the social fallout
A big part of the problem is a lot of people had their brains melted during COVID and became convinced that SARS-CoV-2 was a deliberately engineered bioweapon, so I guess it’s not much of a leap to “AI would make that even easier.” Also, a lot of these AI boosters/enthusiasts think they understand every field outside their own, even better than subject matter experts, and sycophantic LLMs reinforce that notion.
I tend to cringe when a scientist says it is impossible to "practically" make or do such and such. Here I don't include those things that are physically impossible (like, accelerating to the velocity of light, or perpetual machines with no energy input). Time and time again human ingenuity had falsified proclamations of limits to practical attainment.
There is no a priori reason to assume there can be no human virus with 90% lethality. True, there is no such *known* human virus: it could be unknown because such a virus might be rather unfit to propagate through to a pandemic status or because of human genetic diversity. But there are bird flu types (H5N1, H5N8) that are 90-100% lethal to (admittedly genetically nearly homogeneous) poultry populations. There is zero assurance that a virus cannot be engineered, which uses a highly invariant human cell surface protein as a receptor, such as the Na-K ATPase transporter subunit or the primary glucose transporter. Whether a high-school student can make it? I don't know but I never underestimate the younger generations.
Not just 90% lethality. 90% lethality while spreading easily and spreading primarily during the asymptomatic period, so nobody notices they are infected until it's too late. I think such a scenario is extremely unlikely. Maybe not impossible, but sufficiently unlikely that I'd rather worry about other things.
And, in any case, the only thing we can do to protect ourselves from such a virus is to improve public health and invest in rapid vaccine development.
Feels like the biggest threat from AI in relation to viruses is the deleterious effects it can have on public trust and critical thinking.
If we produce a big generation of people with stunted development, incapable of independent thought and lacking the ability to recognise when they’re being exploited, public health campaigns will falter, vaccine rates will plummet, and viruses will have a lot more hosts to evolve in.
Who needs smart programs when you have a dumb populace?
One scenario (which I find implausible) that Yudkowsky describes for AI caused human extinction is the AI creates self-replicating nanobots which get inside and kill everyone on cue. Now, my impression is that we are all under attack by self-replicating nanobots all the time and have been for however many hundreds of millions of years. What are your thoughts on such a scenario?
The best way to evaluate Yudkowsky’s statements is to think of him as a cult leader rather than a scientist. He doesn’t care about plausibility or physical constraints. He only cares about the emotional responses he can evoke.
Basically where my head has been at on the guy! Thanks for the response.
To add, Yudkovsky never completed any formal education nor ever held a job. Which means he never had a person of both greater competence and authority to tell him: your work is crap, go redo it or you get an F / lose your job.
He's basically an ignorant grifter with a gift of gab.
Setting aside the issue of AI-engineered pathogens for a second, I'm currently reading Annie Jacobsen's "Biological War: A Scenario", and I'm wondering how plausible you find that. The Russians apparently have a biological warfare program carried over from the Soviet era, which has weaponized pathogens like smallpox, Marburg, and plague. In her fictional scenario, a Russian scientist feeling guilty tries to destroy the his research and inadvertently ends up freeing pneumonic plague which has been genetically modified to induce euphoria in the early stages to keep the patient active and spreading even when they begin to get symptomatic. This is supposed to be well-researched and based on factual information from defecting Russian scientists. In the scenario, this basically leads to complete society breakdown and mass death. Is this plausible to you?
I’m still curious to hear your response to Soviet/Russian engineered pathogens and how bad you think they might be. Your article seems to portray such scenarios as highly implausible.
Bioweapons research is typically not about making pathogens worse (they are bad enough already), it's about delivery systems and control. Can you somehow infect an entire city at once? Can you infect the enemy while you protect ourself? A smallpox release would be pretty bad though, in particular now that most people are no longer vaccinated. No engineering required.
My main point remains: The stuff that exists naturally is as bad as or worse than anything engineered. Let's worry about naturally occurring pathogens first.
Yeah, I feel like you didn't really read my first comment or address my question. They genetically modified pneumonic plague to make the infected person euphoric during early onset. This addresses exactly what you're talking about. The euphoria makes the infected person continue to be active and social even through the onset of symptoms, increasing spread. They also did gain-of-function modifications to increase the R0.
Again: Do you find this plausible? Both the feasibility of the modifications and that they would wreak far more damage upon release?
Why would the Soviets/Russians go to so much effort to genetically modify these pathogens if naturally occurring ones are as bad or worse. I'm not a virologist, but I can't see how it is possibly the case, and as far as I can tell you have not given any kind of reason why this would be.
Let's start with the fact that pneumonic plague is not caused by a virus.
I don't know anything about this specific research so I can't comment. In general I'm wary of third-party recollections of research because often what is being described is quite different from what the actual research was.
Masks work great against bacteria and so I expect if there was an outbreak of pneumonic plague people would start masking and that would bring the outbreak under control rather quickly. Also, antibiotics work against bacteria, unless you now also throw in genetic engineering for drug resistance.
Would some people die in such an outbreak? Yes. Would it be a world-wide disaster? Probably not.
Yes, but according to Dario Amodei it “will actually be possible to cure most human disease in ~5-10 years” Given the biological knowledge gain implied by this claim, maybe Noah is onto something. Maybe if this claim and others like it turn out to be wrong, then we’ll still be discussing hypotheticals 10 years from now. I expect to be reading hypotheticals (and AI post-mortems), 10 years from now.
Amodei is either delusional or deliberately misleading. I’m not sure which one it is but wouldn’t be surprised if it was the latter.
https://substack.com/@clauswilke/note/c-316736967?utm_source=notes-share-action&r=125478
Probably both.
You seem to suggest that tinkering with the measles virus would be a viable route (to a devastating supervirus), rather than AI wizardry.
If the definition of supervirus is “kills 90% of the human population” (or even just 10%) then no, measles won’t do that. A world-wide measles outbreak would be bad, but it wouldn’t end civilization.
Am I wrong for understanding the pattern of lethality and transmissibility in nature is a consequence of the evolution rather than an inherent trade-off in the virus design?
I can believe there is an inherent trade off in long asymptomaticity and transmissibility as you justified.
None of this really comforts my worry that something resembling a more dangerous version of COVID could be made.
Arguing that in nature, viruses are rarely highly lethal and transmissible means it will be difficult to engineer is analagous to arguing against wheeled locomotion on the grounds that it doesn't occur as a transport method in nature. I think the point is that there are other evolutionary constraints that push nature to be like nature and nature is rarely a good upper bound on "the fastest way to fly", "the most energy efficient way to move" or "the deadliest possible virus".
Am I wrong for believing that this is conceivable: some (much smarter) future AI system use advanced computational modelling, improve the algorithms themselves, to generate 1000 promising virus candidates that are tweaked, more lethal versions of COVID?
The SARS genome is only 30K base pairs (60K bits of information); this is a much smaller search space than say, an image (obviously you know more than I do about the relative difficulty of searching this space but the remaining obstacles really seem down to ample training data and modelling of changes).
Assuming this is biologically possible: could the shear number of viruses prevent effective response, would we be able to stop a misaligned AI from doing this?
I take your point that computational modelling isn't at this stage yet; when, in your opinion, is the correct time to panic, my leading candidate is this: if AI starts making meaningful progress on computational biology, especially if some kind of rapid testing and feedback system is established on physical beings.
Smith is a known tool.
But your argument (however correct it may ultimately be) starts with "I'm a computational biologist and used to evaluate AI systems, and let me tell you MY FIELD IS SO SO SO SO HARD, MAN IT IS SO GODDAMN HARD!!", and this is just so funny in 2026.