The answer has arrived in the form of a video by Brandon Frenz, a biochemist who did his PhD work at the Institute for Protein Design at the University of Washington and who has well over a decade of experience designing proteins and protein binders. In the video, Frenz explains that the main difference in Claude’s pipeline over previous pipelines is the use of ESMFold2 for scoring designed binders. He also explains that ESMFold2 is both faster and more accurate than the competition, and that he himself these days is consistently using ESMFold2 to score protein binders.

The entire video is really good, and I encourage you to watch it. At the beginning, Frenz breaks down what exactly Claude did to design binders, how it had such a high success rate (primarily, by using ESMFold2 to score designs, a choice that was hard-coded into the prompt), and also how it wasted enormous amounts of compute, probably on the order of 100x more than is actually required to generate binders of similar quality. You can spend $20,000 using Claude, or you can spend less than $200 using the appropriate tools directly. In the second half of the video, the Frenz provides a detailed, step-by-step tutorial on how to design peptide binders that pass the same filters Claude used, how to evaluate the designs for potential problems, and so on. It’s a great video.

The video also makes another important point: Yes you could design binders with Claude and $$$, and let Claude handle all the orchestration, software install, and so on. But, you’re probably better off using a platform such as the one Frenz is building, where you have point-and-click access to all the popular design tools. Such platforms didn’t exist even three years ago, but now increasingly there are companies that offer nicely integrated platforms anybody could use, as long as they have access to a well-written tutorial and a couple hundred dollars to pay for compute.

There you go. Whoever wrote the Claude prompt knew what they were doing, or maybe they just got lucky and picked ESMFold2 because AlphaFold3 was out due to licensing requirements. In either case, if you are working on binder design, you can swap out AlphaFold3 for ESMFold2 in your pipeline, for increased throughput and better scoring. So, maybe Claude did make a contribution to protein design after all, if it helped us to realize how important it is to switch over to ESMFold2.

More from Genes, Minds, Machines

To set the stage, let’s first discuss what happened. Anthropic used Claude Science to design a number of peptide binders against several targets. These designs where subsequently tested experimentally by the company Adaptyv. This company provides testing of binders as a service, and it is known for various binder-design competitions. They partnered with Anthropic to compare how well Claude did relative to human protein designers that participated in prior competitions. By all accounts, Claude did very well. Here are some of the key claims, taken from a blog post published on the Adaptyv website:

95% of the designs expressed, which three years ago would have been an impressive headline, showing the rapid progress of AI tools for protein design in recent years. This number matched the best expression rates of our EGFR competition which had hundreds of expert protein designers, and surpassed other challenges such as the RBX1 one. Out of these, 354 of all designs (1,320) bound their target, an overall hit rate of 26.8%, and the per-target hit-rates vary quite widely.

When compared to our competitions, Claude surpasses all their hit rates, especially when looking at every single run for each target Anthropic submitted as in the plot above. For a fair comparison, we have subsetted each competition’s results to only include de novo minibinders. Claude achieved an 80% hit rate on TREM2, greatly improving over the 38.3% we reported in our competition, and even on trickier targets such as 15-PGDH, it has a success rate more than 3-fold higher than observed on Proteinbase.

In addition to the Adaptyv blog post, we also have access to a blog post by Anthropic, a detailed technical report, and a repository with prompts and data. This effort seems to be pretty well documented. And yet I don’t fully understand what exactly Claude did. The prompts folder in the repository contains a file that is over a gigabyte in size. Who knows what is all in there. There is a plain-text prompt file that is 16,000 words of detailed instructions on how to do protein design. This is an amount of material comparable to a PhD thesis on the topic. A competent protein-design expert put all of this together to help Claude along.

Since Claude apparently did better than human protein designers, I’d like to know what exactly enabled Claude’s success. I have a simple principle for evaluating claims of major advances: Can I find a brief explanation, 2–3 sentences, of what the core new idea is that enabled the advance? What exactly is different in this new approach compared to what we have done previously, and how does it lead to better results? Absent such an explanation, I tend to be skeptical, as people are great at confusing themselves. And AI in particular is exceptionally great at finding loopholes, workarounds, or otherwise arriving at a solution without actually doing what we thought the problem statement required.

If I understand correctly, Claude ran existing protein design tools, such as RFdiffusion, ProteinMPNN, ESMFold, Boltzgen, etc. It did not bring anything new to the table in terms of better folding models, better energy functions, or better generative algorithms. So how could it possibly do better than a human expert using those same tools?

There are some possibilities. First, maybe Anthropic threw more compute at the problem than other groups do. There’s a relationship in protein design between the amount of compute spent and the quality of the results obtained. Design is fundamentally a search problem, and if you search longer you’ll get better solutions.2 It looks like Anthropic allocated 2,500 H100 GPU hours per design, which is a lot but also not outrageous. You can buy this amount of compute for about $7,000–$10,000 on the open market (~$3 per one H100 GPU hour).3 For comparison, the recent ESM C paper used 1,500–2500 H100 GPU hours per design,4 and I would estimate most of the leading groups use similar amounts. So compute is not the difference.

Second, it is possible that the field has simply moved forward. New methods for peptide-binder design are released every few months. Maybe Claude took advantage of some tools that either weren’t available when the previous competitions were held or at least weren’t widely used. For example, Claude used FreeBindCraft instead of the regular, highly popular BindCraft, and maybe that version is a bit better. For sure its website claims it’s faster.

Third, all the targets are widely known and have been used in various binder-design competitions. Maybe Claude scanned the literature and found for each target the specific tool and/or parameter settings that performed best in prior design efforts. Or maybe Claude filtered its own designs on the basis of similarity to known successful designs.5 Alternatively, it is possible that Claude scanned the existing literature and discovered the overall best current design pipeline and used that consistently.6

Fourth, there may be a component of luck or survivorship bias. Claude made some choices about what tools to run and with what settings and some of those choices may simply have been lucky. If Claude’s design attempts hadn’t been successful we wouldn’t be talking about them.

It is important to emphasize what Claude has not done. It has not done any actual science that moves the protein-design field forward. For example, it has not tried different binder-design platforms to figure out which has the highest success rate in subsequent experimental testing. It has not tweaked design parameters and synthesized the resulting peptides to figure out which parameters lead to toxic peptides and which do not. All it has done is one-shotting the solution. Press the button, Claude spins up a few GPUs, and out come some novel peptide binders that somehow work. This is great, but Claude has not actually learned anything new about binder design. It could not, by construction.7 Whatever it has done was already present in the literature it processed as part of its “reasoning” process.

Now, on the flip side, I want to highlight that there is some value in having Claude run protein-design tools. Figuring out these tools and getting them to run is not a trivial task. It takes a lot of experience, and also it’s not fun to fiddle with python environments, CUDA incompatibilities, outdated dependencies, and so on, just to install all the latest methods. If Claude can sort this out on its own that sounds appealing to me. Now I suspect that an experienced protein designer will still prefer to run the tools manually, if only to have complete control, to be able to tweak things, and to know exactly which methods are used and how. But, I can see plenty of use cases where outsourcing this to Claude may be worthwhile. In this LinkedIn post, Brian Weitzner estimates there are fewer than 1000 people total that can do this kind of work. Many companies with protein-design needs may not be able to hire one of these people, and in particular not one of the much rarer experts who truly understand how things work and who are moving the field forward. If companies can instead throw some money at Claude to get useful designs, that may be a solution that works for them.

And, for no good reason, I’ll close by pointing out that no angry teenager will use Claude to design novel peptide binders, because no teenager has $50k–$100k lying around to pay for the required GPU time.

Update: Apparently what gave Claude the edge was access to ESMFold2, which has only recently been released and is not yet widely used. See here for details.

More from Genes, Minds, Machines

The article opens with a fictitious story about a disgruntled teenager in 20291 who asks an AI to design some lethal viruses, which he then deploys and ends up killing 90% of the human population.

Immediately following this story, Smith writes: “I have yet to hear an even halfway-convincing argument as to why this scenario is far-fetched.” Pro-tip: When you find yourself writing such a statement about a topic you’re not an expert in, there are only two possibilities: 1. You are not talking to the right people. 2. You are talking to the right people but you’re not listening to what they’re saying. On that note: The article is paywalled. I can’t read beyond the first few introductory paragraphs. I also can’t post comments. All I can do is write on my own blog, making some assumptions about what the article may say. It is possible that every point I’m making here is competently addressed and strongly refuted in Smith’s article. If that’s the case, more power to him, and I’ll stand corrected.

Who am I? Why should you listen to me? Well, for one, unlike Noah Smith, I have actually computationally designed viruses and other biological systems.2 Also, my day job involves developing and evaluating AI systems for protein and peptide design. So I know a thing or two about state-of-the-art AI tools in biology and about the challenges of designing and building functional biological systems. I also know a bit of virology.

Computational design of biological systems is unfathomably difficult. Experts who have dedicated their life to this topic routinely hit their head against the wall when nothing they try seems to work. PhD students in 2026 using state-of-the-art AI software are spending months or years trying to design simple peptide binders that inhibit some enzyme or pull down some protein, and the majority of their designs fail, or don’t express, or are toxic.3 But in Smith’s fictitious world a disgruntled teenager with no special training in biology can just solve a problem thousands of times more complicated than designing a peptide binder. The distance between where we are today and where we would have to be for Smith’s story to have any realism is enormous. And then, even if you could design the perfect virus, assembling and distributing it would be a non-trivial task in its own right. You don’t just order a working virus from temu.com. See for example this article by .

I’m not discounting the possibility that at some point somebody may cause some harm with an AI-designed biological agent. But we need to consider scale. A disgruntled teenager could also get their hands on a fertilizer bomb and blow up a busy shopping mall. No AI needed. So damage on the scale of maybe a couple hundred to a couple thousand people dead, while devastating for the individuals affected, is not an existential threat. The question is whether a much larger attack is possible. Something that would kill millions of people.4 I am quite confident this is not something we need to worry about, at least not for another few decades.5

Nobody knows how to build a supervirus

Let’s set aside all the issues of whether AI can design viruses, how hard it may be to then build them, and finally to distribute them to unsuspecting victims. Let’s assume all of these issues are solved.6 So, you’re armed with your supervirus-generating AI and your rogue lab that will synthesize and weaponize viruses for you. What properties do you want the viruses to have?

“Well, duh,” you say, “I want them to be lethal!” Ok, here you go: Ebola virus, hantavirus, rabies virus. All extremely lethal. And yet, we’re not really that worried about them. At least we don’t think of them as civilization-ending superviruses. Why is that?

Hantavirus doesn’t spread human-to-human.7 Don’t inhale infected rat droppings and you won’t get it. Similarly, rabies doesn’t transmit human-to-human, primarily because once people are becoming contagious they are so incapacitated that they’re unlikely to bite you. Now Ebola is different, it is actually quite contagious, but it requires contact with the bodily fluids of an infected person and that contact is easily avoided unless you’re a direct caregiver.

“Ok then,” you may say, “I want them to spread through the air!” That’s fine, plenty of viruses spread easily, but there’s generally a tradeoff between how easy a virus spreads and how lethal it is. For a virus to spread easily, the infected patient needs to shed a lot of viral particles, and this requires a high viral load. But, the patient needs to be able to walk around and function while experiencing a high viral load, otherwise they won’t spread the virus. This generally means the virus is not that deadly. An example of a virus that spreads extremely well is measles virus. The virus can remain suspended in the air for up to two hours. If you enter a room in which two hours prior somebody with measles spent 15 minutes you may you catch the disease. But, measles kills “only” about one in a thousand infected people. It’s incredibly contagious, but it’s not that deadly. It’s really bad though. We’ll get back to it later.

“I got it now,” you may say, “I want viruses that have a long incubation period, during which they can get transmitted, and then later I want the infection to turn really bad and kill the patient.” Ok, fine, such a thing would probably deserve the label “supervirus.”8 I’m not sure what the biological mechanism would be though. How do you build this? Viruses with long incubation periods are typically not contagious during the asymptomatic period. The may integrate into the host genome and lay dormant for a while, only to reactivate later (e.g., varicella-zoster virus, the causative agent of chickenpox and shingles). Or they slowly make their way through your nervous system (e.g., rabies virus). The one virus that sort of has the properties we’re looking for is HIV. Extremely long incubation period. Patients can be contagious without showing obvious symptoms. Nearly 100% lethal without treatment. But, not easily transmitted. We commonly see this tradeoff. Viruses that hide in specific compartments of the body for long periods of time will not usually transmit easily. I’m not aware of any virus that has even remotely similar properties to HIV but spreads like a respiratory virus.

In general, there are tradeoffs between how lethal a virus is and how easily it spreads. Consider the common flu (H3N2 influenza) and avian influenza (in particular, H5N1). H3N2 influenza transmits easily person-to-person because it targets a receptor that is located in the upper respiratory tract. H5N1 influenza, by contrast, targets a receptor that is located in the lower lungs. This causes more severe disease but limits spread. Epidemiologists have reasonable concerns about a potential H5N1 pandemic, but to date the tradeoff has held. We have not seen a civilization-ending H5N1 pandemic.

There is simply no good reason to believe that AI will magically be able to work around these fundamental tradeoffs of biology, that it could somehow design a virus with the lethality of H5N1 influenza but the contagiousness of H3N2. The basic science required to build such a thing has not been done. When people believe that AI could figure this out on its own, without first running extensive and expensive basic science experiments, they have left the realm of science and have entered what can only be described as religion. They hold a strong belief not supported by any evidence or causal chain of logic.

The viruses causing the most damage are not what you think they are

As I just said, a virus that first spreads silently throughout the population and then flips a switch and kills everybody is science fiction. It does not exist and will never exist. And among the viruses that do exist, the ones that seem the scariest are not necessarily the ones causing the most damage. For any virus that does or could exist, we have to consider how an outbreak would affect and interact with human behavior.

Consider the spectrum of possible viruses, from mild to highly lethal. On one end you have something like the common cold, which spreads easily and infects millions of people every year. It rarely causes serious illness so people mostly just ignore it and live their regular lives. The total damage caused by the common cold, in terms of morbidity and mortality, is not that large because the infection is rarely lethal. On the other end of the spectrum, you have something like Ebola, with mortality rate up to 90%.9 Ebola is a scary virus. It is a killer. Yet the total number of people killed by Ebola is not that high either. Getting infected with Ebola is generally a death sentence, and therefore people take it seriously and do their best to avoid transmission. Ebola will never turn into a world-wide pandemic. If Ebola prevalence kept rising, at some point people would simply isolate, stop interacting with the world, and wait it out. The viruses causing the most damage are not found on either end of the lethality spectrum, they live in the middle.

There is a Goldilocks zone of lethality for a pandemic virus that will cause maximum mortality, and it’s a level of lethality that is much lower than you may think.10 In fact, I think COVID was close to ideal in terms of a virus causing global morbidity and mortality. It caused sufficiently severe infections that many millions globally died, and yet its infections were also often sufficiently mild that people could just dismiss the virus and pretend it wasn’t an issue, thus contributing to further viral spread. Notably, COVID also had the property of asymptomatic transmission, where some people spread the virus without themselves feeling particularly sick or showing any symptoms. It was near perfect as a pandemic supervirus.11

So, if you’re worried about AI-designed superviruses, think COVID, not Ebola. And if you’re now a little less worried, maybe you understand where I’m coming from.12 Yes, COVID was bad. But it did not end civilization. We’re still here. At least those of us who survived. If somebody actually designed a virus with AI, which I still maintain is virtually impossible, I doubt they could do more damage than COVID did.

The risk from nature is still worse

What really bothers me about these stories about how AI will design killer viruses is that they are pure fear mongering, and more importantly they divert attention from the things that matter. There are deadly viruses all around us, and viruses routinely jump from animals to humans and cause severe outbreaks. This is a real, documented risk. This is the risk we should worry about. This is the risk we should prepare for. But instead, we worry about fictitious AI-designed viruses while ignoring the real viruses that stare us in the face.

Let’s go back to measles. It is bad. Really bad. If somebody gave themselves the goal of designing a supervirus and came up with something like the measles we’d probably call that a success. Measles is incredibly contagious. On average, every infected person infects over ten other people. Around 10-20% of measles cases result in hospitalization, and about one in a thousand results in death. And if you survive the measles, you may experience permanent vision loss, hearing loss, brain damage, intellectual disability, or a destroyed immune system. If you’re really unlucky, 10–20 years after your measles infection you develop subacute sclerosing panencephalitis, which is an awful, progressive brain disease that slowly turns your brain into mush and eventually kills you.

Now here’s a real risk involving measles that I sometimes worry about. What if the measles virus experienced a set of immune escape mutations where suddenly existing vaccines no longer work? If this happened the consequences would be horrifying. We’d have an immediate, world-wide measles epidemic long before we could develop and roll out a new vaccine. What’s the likelihood this will happen? Informed estimates are that it is low. But realistically speaking, even if the likelihood is low, it’s probably higher than the likelihood that a disgruntled teenager will design a supervirus with AI. So, something to think about.

More generally, it feels to me like society is currently not particularly concerned about real viruses causing real devastation. There’s a massive Ebola outbreak right now in the Congo. Nobody cares. Measles are surging in the US. Nobody cares. HIV is running rampant in Sub-Saharan Africa. Nobody cares. And for sure nobody cares anymore about COVID. So spare me your concerns about fictitious, AI-generated superviruses.

What protects us from deadly viruses, regardless of their origin, is good public health practices and basic research. Ongoing surveillance to catch emerging pathogens early. Consistent vaccination against known pathogens so they are kept in check. Research into rapid vaccine development and into novel antivirals. Investments into improved indoor air quality and air filtering. Consistent hand washing. But this stuff is boring and doesn’t sell Substack subscriptions. Let’s be real. Fear mongering about AI superviruses is a much better strategy to make money online.

More from Genes, Minds, Machines

By performing a systematic benchmarking study, Berry et al. find that pLMs consistently do poorly in identifying mutations that enable or increase novel function. My own lab had recently made a similar observation, so this part of the study was not that surprising to me. What comes next is more important, however. Berry et al. show that an alternative model performs much better. What is the alternative model? It is a position-specific scoring matrix (a PSSM, this is basically just site-wise amino-acid frequencies in a multiple sequence alignment) minus a pLM. So, to find variants that generate or improve novel function, we need to look for mutations that are common in multiple sequence alignments, and then among those pick the ones that the pLM thinks are bad. Yes, bad. The pLM score is subtracted, so a high score from the pLM means the mutation is not a good candidate for further exploration.2 This is exactly the opposite of what everybody else in the field has done to date.

One of the most impressive examples provided by Berry et al. is the case of DraNramp, a protein used by bacteria for manganese (Mn²⁺) and iron (Fe²⁺) uptake. In nature, the protein does not transport magnesium (Mg²⁺), but many laboratory variants are known that can perform this function. So, how well do pLMs such as ESM-1v do at predicting variants that enable magnesium uptake? Terribly. Nearly all of the proposed variants excel at importing manganese, and none can import magnesium (Figure 2A). But, when Berry et al. use scores from the PSSM minus the pLM, they recover many variants that are quite good at magnesium uptake (Figure 2B). The pLM likes to maintain the current function, whereas the PSSM is able to explore new functions.

What is going on here reminds me of the old Jesse Bloom work showing that function-enhancing mutations are often destabilizing to the protein, and therefore more stable proteins are better starting points for the discovery of beneficial mutations.3 Mutations proposed by pLMs are in effect mutations that predominantly maintain or increase protein stability. They fit perfectly into the provided sequence context. The flip-side of this constraint is that they are unlikely to lead to functional improvements. By contrast, the context-free, site-wise models simply propose mutations that are common; because these mutations were chosen without considering sequence context they may decrease proteins stability or otherwise mess things up. But this messing things up may be exactly what we need when we’re looking for new or improved function.

One aspect of the paper that I find unnecessarily confusing is that it conflates sequence context with function. Their Figure 64 suggests that in natural sequences, we see different sequence contexts because they correspond to different functions.5 Consequently, to sample mutations that enable these various functions, we need to somehow break out of the sequence context, and a context-free model does exactly that. The problem that I have with this explanation is that new-to-nature functions cannot be found among any of the natural sequences, by definition. And yet, the natural sequences may well contain mutations that, in the right context, can provide new-to-nature function. The DraNramp example highlights this possibility. Natural DraNramp variants cannot perform magnesium import.6 And yet, they contain mutations that, in the right context, can import magnesium.

In fact, we don’t need to assume different functions in natural sequences. We just need different sequence contexts, which arise for example because of covariation among sites that are in physical contact in the folded protein. We know that contacts create strong evolutionary constraints among sites, visible in covariation in multiple sequence alignments. These constraints are so strong that covariation can be used to identify physical contacts from multiple-sequence alignments. And pLMs have learned this covariation, which we can tell from the fact that we can also use them to infer contacts.7 In other words, pLMs are great at taking into account sequence context. They are literally trained to fill in the blanks given the surrounding context.8 Combine this with reasoning along the lines of Bloom et al., and we should not be surprised that pLMs have a tendency to propose the most conservative, stabilizing, function-preserving mutations, and this tendency goes exactly opposite to what we want when we’re looking for novel function.

Were does all of this leave us? First, the method proposed by Berry et al. is simple to implement. Anybody can use it. So that’s great, we have a new arrow in our quiver. Second, we’ve still only scratched the surface with respect to understanding what pLMs are actually good for. I have no doubt that they are an amazing tool that will have profound implications for the future of protein science. However, this does not mean that these models are currently used appropriately. Clearly, zero-shot predictions from pLMs are not that useful, in particular not if the goal is to find variants that provide novel function. But other pLM applications are totally legit, such as homology search or contact prediction. I’m looking forward to discovering more about how these models work.

More from Genes, Minds, Machines

The one thing you must never do is ask what the error bars actually represent. There be dragons. Ask five different scientists what quantity you should use to generate the error bars and you’ll get five different answers: Plot the standard deviation; plot the standard error; plot the 95% confidence interval; plot the range; perform a Bayesian analysis and plot the 95% credible interval. The list of possibilities goes on and on.

The dirty truth of error bars is there is no generally accepted quantity they should represent. You can plot whatever you want. I personally think that the standard error is the right choice, but I’ve been in this business long enough to know that what I think is irrelevant. Plenty of people do not plot the standard error, or would be hard pressed to even explain what the standard error is. If you see error bars and you think “standard error” you’ve spent too much time in stats class and not enough time in the real world. People chose an experimental science because they weren’t that excited about stats. Do you really expect them to remember the definition of the standard error?1

The most common mistake I see is people plot the standard deviation instead of the standard error. How is this a mistake, when I just said “you can plot whatever you want”? It’s a mistake in the sense that standard deviation does not represent an error. So if you intend to show an error, and you plot the standard deviation, you have not achieved your stated goal. It’s shocking how often I’ve talked to even senior scientists who were confused about this topic. These days, if somebody is not a professional statistician, I’ll just assume they don’t know the difference between standard deviation and standard error.2

But even if people do understand that plotting the standard deviation is not a good idea, they may plot something other than the standard error. The most common reasonable choice is to plot a confidence interval instead of the standard error. But then of course one has to pick the confidence level, and while 95% is probably the most common one others can be used and are valid.3 In general, because there are so many possibilities of what error bars can represent, any visualization with error bars should always be accompanied by a statement of what quantity is shown. So pay attention to the literature you read. Do the authors specify what they are plotting as error bars? In my experience, in the vast majority of cases they don’t, and nobody asks. It’s a cargo cult. The field wants to see error bars, but it’s comfortable not knowing how they were generated or what they mean.

Now lets talk for a moment about curve fits, and in particular about non-linear curve fits, where the conventional visualization employs a confidence band to show uncertainty. Everybody makes them. You can see these visualizations all over the place.4 But have you ever stopped to ponder what they mean? I am quite confident that there are very few people who can correctly explain what the confidence band of a non-linear curve fit represents.5

Let’s proceed step by step. First we consider a linear curve fit. There, the confidence band takes on the characteristic shape of an hourglass. This shape arises because both the slope and the intercept of the line have an error. You have to imagine the curve both moving up and down and rotating, and that combined movement sweeps out an hourglass.6

For a non-linear fit, the intuition gained from linear fits is not that useful. While the mathematics of linear and non-linear confidence bands is the same, the visual effect is completely different. Mathematically, every parameter in the fit has an error, and to explore the error surface means to find all the parameter combinations that have high probability density. Then you calculate where these parameter combinations place the response variable as a function of the predictor, and the boundary of that region defines the confidence band. This is the same for linear and non-linear fits. Because a linear fit never changes shape (it’s always a straight line), for linear fits we only have to imagine the line moving up and down and rotating. But for non-linear fits, as you change multiple parameters the shape of the curve changes as well, not just the location, and so the curve wiggles around as it sweeps out the confidence band.

The consequence is that individual fits are much more wiggly than the confidence band suggests. The confidence band tells you that the fit would likely run somewhere in this region, but it doesn’t tell you what kind of path it would take. I have explained this hundreds of times and I still don’t quite know what to make of it. I usually tell people to not plot confidence bands on non-linear curve fits.7

In summary, while I’m not arguing that error bars don’t matter, I do think they are taken too seriously. There are occasions where not plotting error bars or confidence bands is a perfectly reasonable choice to reduce visual clutter. Either way, when you plot error bars make sure you always specify what they represent, and when you review other people’s work check whether they report what they are plotting. Finally, pay attention to whether people are confusing standard deviation and standard error, and if they do gently explain to them what a sampling distribution is.

More from Genes, Minds, Machines

Since I am also a person of the internet, I am also losing my mind. Who is Nene Royal? Her guitar playing is next level,1 and her singing is outstanding. How can she be so good, at such a young age? Is she some sort of genetically engineered child prodigy daughter of music professors who coached her since she was a toddler?And why did nobody know about her before her AGT performance? Walking onto the stage, she looks so meek and reserved, but then she transforms into a complete rockstar. How is this possible?

Here, I’ll try to answer these questions. Let’s start with the third one, her family background. Surprisingly, nobody in her family is a musician. She started playing the guitar at age 7 because at school, by accident, she ended up in a music club. She originally wanted to register for the computer club but on the day of registration she overslept and arrived late. So her desired club was already full and the only remaining option was music. She signed up for that club even though she didn’t know anything about music at all. Then she needed to pick an instrument, and the only option left was guitar. So that’s what she chose.2

Next she needed to buy a guitar. Neither she nor her father knew anything about guitars. They were told it had to be an acoustic guitar but they had no idea what that meant. Her father says he didn’t know what the difference was between an acoustic and an electric guitar, nor where to buy one.3 He eventually bought a cheap acoustic guitar for 1000 baht (roughly $30) at a store that sold school supplies. But this was good enough to get started, and Nene fell in love with playing the guitar and making music.4

Now let’s talk about her stage presence. You can see in the AGT performance that she interacts with the audience as if she had done this a thousand times. And the truth is she has. Nene’s family is not rich, and her parents couldn’t afford buying her expensive instruments or paying for music instruction. Her father came up with a solution: Nene could work for tips as a street performer. She started doing this when she was only 9 years old, and she has continued doing this ever since (with a brief interruption during COVID). For years now she performs 2–3 times a week at a market in Phuket, Thailand. She must have close to a thousand shows under her belt in front of a live audience. And these shows are long, 90 minutes or more at a time. This is the experience level of a seasoned, professional musician. It’s not surprising that she’s so comfortable on stage and knows how to work a crowd. By the way, many of her shows are streamed on YouTube. Here is an example from December 2023:

This brings us to the question of how come nobody knew about her before AGT? Well obviously, that’s not true. As far as I can tell, she had a few hundred thousand subscribers on YouTube before her AGT performance. Also, on YouTube you can find older interviews with her and reactions to her videos that predate AGT.5 Notably, Brian May, the guitarist of Queen, praised her cover of Bohemian Rhapsody in early 2026. Between her own YouTube channel and the channel of her band Ozone, there are almost 2000 videos of her on YouTube, and her channel has an accumulated 265 million views. But the world is a large place, and somebody can be known to hundreds of thousands or even millions of people and still be rather obscure on the global stage.

Finally, how can she be so good, in particular at such a young age? Honestly, that’s the point where the explanations fail. It simply shouldn’t be possible. The best explanation I can give is she’s a once-in-a-generation talent combined with exceptional work ethic and a supportive family. Music is complicated. It usually takes a decade or more of dedicated practice under the guidance of an experienced teacher to become a world-class musician. Normally, when you see a musical child prodigy, they come from a family of musicians and have had the most sophisticated musical instruction from the moment they could walk. The fact that Nene does not have this background, and is mostly self-taught from watching YouTube, is just mind-blowing.6

And the speed at which she developed her capabilities is simply beyond comprehension. She got her first electric guitar at the age of 9, and by age 10 she was already playing some of the most difficult songs, songs that for many guitar players remain permanently aspirational. And not only did she play those songs, she totally owned them. Check out the video below, where she’s “just practising.” The song is Through the Fire and Flames by DragonForce, deliberately written to be difficult to play. A 10-year old should not be able to play like this.7 At that age, they lack the precision of movement, the strength, even just the hand size, to play at such an advanced level. And yet she plays it with ease. Her timing is spot on, her sound is amazing, and she doesn’t seem to struggle in any way.8

So clearly, she has a special talent, not just a special mind for music but also exceptional bilateral coordination, the strength and dexterity required to play extremely difficult guitar parts, and exceptional fine motor skills to be able to play at high speed in time and with great sound. Normally, when you hear children play, even talented children with good coaching, their play has a certain child-like quality that they eventually grow out of. It appears Nene had grown out of this by age ten.

While most of the videos of her show her covering well-known hits, Nene has also started to write her own pieces. There is an instrumental of hers called “Real Verse.” With this piece she again demonstrates she’s an absolute master of the electric guitar. I’m looking forward to more of her original compositions, which will certainly be forthcoming.

If you’ve made it this far you may wonder whether all she can play is metal. The answer is no. She has amazing breadth. Check out her video covering “Married to the blues” for great soulful blues singing (and guitar playing, obviously). There are also videos of her singing Billie Eilish. Oh, by the way, when she performs that song she plays the piano…

Throughout this post, I have drawn heavily from a podcast interview with Nene and her father. The interview is in Thai but English subtitles are available.

This was an unusual post for this blog. I’m sorry, I got carried away and spent way too much time the last week watching Nene Royal videos. I post about what I find interesting, and I get interested by weird stuff.9 There’ll be more biology and AI soon.

I believe both of these responses are misguided. But also, they highlight how important it is to cover this material in detail. The average student in math, computer science, physics, biology, or engineering is never going to learn about storytelling, and yet we expect them to give presentations that somehow do not put the audience to sleep. I care deeply about this topic because I’ve sat through too many boring presentations. And it’s particularly frustrating when the presenter has good material in principle but just totally messes up the presentation. That’s why I keep writing about this topic. If only one person gives better presentations after reading one of my posts I have had a positive impact in the world.

Emotions

Let’s start with emotions. Why should you care about the emotional state of your audience? You’re talking about facts, logic, observations. None of this has anything to do with emotions. So it’s not something you have to think about, right?

This is one of the most common misconceptions by people working in technical fields. In fact, even if they’re not consciously aware of it, technical people routinely take actions based on their emotions. And then they employ logic to justify retroactively why they did what they did. Let me provide an example where emotions unexpectedly crop up. If you’re a scientist, you’ve probably had a grant proposal rejected. If so, I bet the rejection contained the following sentence: “There was low enthusiasm because of X, Y, and Z.”3 This is a statement about an emotion. Enthusiasm is an emotion.4 I don’t know about you, but I want enthusiasm in my audience. I want them to jump out of their seats and scream “Yes, YES, this, more of this!”5 And to get this reaction, just logic and facts won’t cut it. We have to reach deeper.

As another example, let’s assume you’re giving a presentation. Your audience will always feel some emotions, regardless of what you say or do.6 To start out, they will bring some emotional baseline into the room. Maybe they are happy because earlier in the day they experienced a major success. Or they are sad or anxious because a loved one has fallen ill. Or they are bored because their job isn’t meaningful to them and they feel they’re just sitting around wasting their time. Whatever their baseline, it will influence how they perceive your talk. So you have two options: You can either try to control their emotional state and guide them to a point where you would like them to be, or you can leave it up to them to experience whatever they feel in response to what you’re saying. To me, the second option is equivalent to saying that if people are bored out of their mind it’s their fault for not appreciating your intellect.7 I would not recommend adopting this attitude, but you do you.

Storytelling

Now why should you care about storytelling? You should care because it’s your primary method of influencing your audience’s emotional state. The story arc is what enables you to take your audience from wherever they are at the beginning and guide them towards the emotional state you want. The better of a storyteller you are, the further you can move your audience. People enjoy a good story exactly because it allows them to forget their own emotional baseline for a moment.

But what about the concerns that storytelling is equivalent to manipulating the audience or making things up, just to tell a story? First, I never said you should make things up. I said a story is “a collection of observations, facts, or events presented in a specific order such that they create an emotional reaction.” None of the observations, facts, or events should be made up, unless you’re explicitly presenting fiction. I’m not asking you to say anything that isn’t true. Neither am I asking you to just bend the truth a bit. I’m simply encouraging you to present your material in such a way that your audience can connect with it.

Second, yes, a good storyteller can manipulate their audience, but doing so is a conscious choice, not an unavoidable consequence of telling a story. Any useful skill can be used for malicious purposes. That is not an argument against developing new skills. Imagine you were telling me you are learning how to cook and my response was that most good-tasting recipes contain too much fat and sugar and people who cook well just want their family to get fat. Would you find that convincing? Or would you maybe object that the better you can cook the more you can prepare healthy meals that keep your family fit? It’s the same with storytelling. If you know how stories are constructed and why they work, you can tell an exciting story that does not manipulate your audience. And as a bonus, you will also get better at recognizing when somebody else is trying to manipulate you.

Summary

Regardless of how interesting and logically sound your work is, you will have to craft it into an engaging story if you want to reach a larger audience. The purpose of storytelling is not to manipulate. It is to overcome your audience’s inertia, most importantly boredom and disinterest. If your audience is bored, they won’t put in the effort to discover why what you’re doing is actually interesting. They simply won’t care. So, you have to do everything you can to make them care. And that will in general involve telling an interesting story.

More from Genes, Minds, Machines

In a recent preprint from my lab (Woolley et al. 2026), we provide a potential explanation. Most importantly, zero-shot predictions have a conceptual limitation that I don’t commonly see acknowledged: Zero-shot predictions cannot simultaneously capture multiple dimensions of a protein’s fitness landscape (Figure 1). By definition, a zero-shot prediction is a single prediction (“make this mutation at this site”), regardless of the phenotype of interest. But what if a mutation improves one phenotype and worsens another? For example, a mutation could increase enzymatic activity and decrease stability. Such a mutation might be exactly what you need if you’re interested in increasing enzymatic activity, but it would be counterproductive if you’re instead interested in increasing stability. Zero-shot predictions do not know what you’re interested in. They just make a guess and hope it’s right. And often it’s not.

I want to emphasize that this limitation of zero-shot predictions is fundamental. It applies to any possible model. Often in machine learning and AI, when model predictions aren’t good enough, we immediately blame the model. “Predictions from this model aren’t that great,” we may say, “but surely we can train a better model that will have the performance we need.” In fact, this reasoning is one of the motivators for large-scale benchmarking projects such as the ProteinGym. If poor performance was primarily due to models being bad, then it’d makes sense to benchmark all the available models and try to find the best one. Unfortunately, when you’re dealing with a fundamental limitation that equally applies to all models, even the best ones won’t be that great. And in fact, we have found that the available models all perform roughly equally well, and that there is more variation among datasets within models than there is between models (Woolley et al. 2026). In other words, whether a zero-shot model works well with your specific system of interest is mostly due to chance. It may work great, or it may not work at all. It’s difficult to predict what the result will be.

But zero-shot predictions do predict something. So what is it that they predict, and why is it that performance in large-scale benchmarks seems to be quite good? At their core, all zero-shot methods work the same way, regardless of the specific model architecture and training data used. These models are trained on a large corpus of available protein data—either protein sequences or protein structures or both—and then make predictions that are consistent with this corpus of training data. Since the vast majority of sequences or structures in the training data represent extant, viable proteins, these models have therefore learned the universe of naturally occurring, viable proteins. They can predict whether a specific mutation is likely going to lead to a viable protein or not. But they cannot typically predict a mutation’s effect on a specific function.

If zero-shot predictions primarily capture viability, we would expect this to be reflected in large-scale benchmarks. And indeed this is the case. Correlations between fitness and zero-shot predictions are consistently higher when considering both fit and unfit mutations than when only considering the highly fit mutations. (See Figure 2 for an example; in our analysis we verified this pattern across over 200 different datasets.) In other words, zero-shot predictions can separate fit from unfit mutations, but they cannot differentiate among the fit mutations. This explains why overall performance in large-scale benchmarks looks good. Across millions of mutations, the models are doing quite well, because the datasets contain both viable and non-viable mutations in roughly equal proportions. This balance leads to decent correlation coefficients between prediction and measurement.

But when analyzing specifically function-enhancing mutations, in particular in “new-to-nature” engineering experiments where proteins are engineered to perform a function they don’t perform naturally, we found that zero-shot predictions were systematically uncorrelated (or even weakly anti-correlated) with the measured effects of the mutations. Zero-shot predictions were not able to identify mutations that would increase function, in particular when the function of interest was something new that would not have been observed in natural sequences.

In summary, zero-shot predictions capture protein viability, but they are rarely useful to identify function-enhancing mutations. This is a generic property of the zero-shot approach, and it applies broadly across different models and model architectures. Model performance in large-scale benchmarks appears to be good because benchmarks assess aggregate performance across millions of mutations, and mutant viability is the primary source of variation across these large-scale datasets. Zero-shot predictions can be useful, in particular to screen out strongly deleterious mutations, but they will rarely point towards increased function in targeted protein-engineering campaigns.

Read the complete study here:
Phillip R. Woolley, Aaron L. Feller, Andrew D. Ellington, Claus O. Wilke (2026) Overestimating zero-shot fitness prediction: Broad benchmarks mask local failures and practical limitations. bioRxiv. https://doi.org/10.64898/2026.06.04.730121

More from Genes, Minds, Machines

I encountered this type of advice most recently in an article in Nature Reviews Cancer on the topic of giving presentations.1 In the first subsection, which covers the overall presentation structure, the article states:

Develop a ‘story’ by ordering your points – working backwards from the point you want to get across to the background required to build up to it – so that the audience has the best chance of understanding what you have to say.—Itai Yanai, Nature Reviews Cancer 2026.

Note how this excerpt doesn’t actually explain how to tell a story, other than that there should be some sort of order from beginning to end. Further, note how the word “story” is in quotes. I take this to mean the authors isn’t sure himself of what a story is. He doesn’t seem comfortable to plainly recommend you to tell a story. He can only recommend you to tell a ‘story.’2

Surely we can do better. Let’s start with a definition. What is a story? A story is a collection of observations, facts, or events presented in a specific order such that they create an emotional reaction.3 The most important element here is the last part, “emotional reaction.” You want your audience to experience an emotion, such as joy, surprise, satisfaction, pleasure, sadness, anger, or horror.4 Which specific emotion you choose depends on your goals, audience, and settings. In most scientific or professional settings, you may prefer a positive emotion (joy, surprise, satisfaction, pleasure) over a negative one (sadness, anger, horror). However, negative emotions have their place. For example, if you want to convince your audience to donate or otherwise provide funds for a specific cause, negative emotions can be strong motivators.

So now we know what our goal is (evoke an emotion), but we still don’t know how to get there. Some people have an innate instinct for story telling; they can create strong emotions in their audience without necessarily knowing how or why. But many people do not. They need to follow a more systematic and structured approach.

I will explain this structured approach momentarily. But before we go there, I want you to remember this one concept: Challenge – Resolution. Every story requires some sort of challenge that then leads to a resolution. The tension between the challenge and the resolution creates the emotional reaction in your audience. If you keep this concept in the back of your mind as you develop your narratives you’ll do better than most.

There are a few different ways of structuring a story, and it is generally helpful to be aware of the various structures and their respective pros and cons. This is material that I teach routinely in my class on data visualization. The concepts I am going to describe are quite general, and they apply just the same when giving a talk or writing articles, proposals, or emails, or otherwise communicating in any way with fellow human beings.

Let’s begin with the most standard, conventional story structure. It has four components:

Here, Challenge and Resolution are the most important. Get through the opening quickly to present the challenge before your audience loses interest. Then spend some time describing the action that gets you from the challenge to the resolution. Finally, present the resolution, making sure that it fully addresses the original challenge.5 Audiences are quite sensitive to challenges that weren’t fully resolved. They will typically describe such situations using words such as “the ending was disappointing.” They may not know exactly why they feel this way, but they know that something was off. Don’t put your audience into this position.

When I teach this material in my class on data visualization, I usually tell my own story of how I ended up writing a book on data visualization and then started teaching the material.6 This allows me to highlight how different elements of the story map onto the various components of the story structure:

Opening: I am primarily a computational biologist, running an active research lab. This work frequently requires creating data visualizations. Unfortunately, many of the new students joining my lab don’t know much about how to visualize data.

Challenge: I spend a lot of time coaching students how to make good visualizations. This means I find myself teaching the same concepts over and over.

Action: I had the idea that I could make my life easier by writing a book about data visualization. If the book captures all my thoughts and knowledge about the topic, then students can just read the book and self-study. They will no longer require my direct coaching, and I can spend more time doing computational biology.

Resolution: The book is written. It has been a great success. However, now I teach data visualization as an official university course, so I still keep teaching the same concepts over and over.

Next is the action-movie structure. It has five components:

The main difference to the basic story structure is that at the beginning the audience is disoriented but hopefully drawn in by the action. Where the basic story structure takes the audience by their hand and gently leads them to the challenge, the action-movie structure does the opposite. It drops the audience in at the deep end and hopes they can swim. This story structure is routinely used in action movies (hence its name), but it is also commonly used in novels. If you’ve ever read a novel that at the beginning had you utterly confused about what was happening, chances are it was using this structure.7 Importantly, when you’re using the action-movie structure, even though none of the five components is explicitly called “challenge” you still need to set up a challenge, either in background or in development. Climax and ending will provide the resolution.

Here is my story in the action-movie format:

Action: In May 2017, I embark on a major project: I write a book on data visualization.

Background: I do this because we need to make many data visualizations in my lab, and I end up telling my students the same things over and over.

Development: Writing the book takes me almost two years; along the way, I add many features to ggplot2 and become a member of the ggplot2 team.

Climax: The book is released in April 2019.

Ending: Now I’m a dataviz teacher.

The final story structure is primarily used in newspaper articles. It has only two components:

The lead will typically contain both the challenge and the resolution, giving away the entirety of the story in the first few sentences. The development part then provides additional material and further detail.

Here is my story in this format:

Lead: Because I see the need for more education in data visualization, I have written an entire book about the topic.

Development: I had previously written an R package to improve figure design, but it wasn’t sufficient: Good judgement cannot be automated.

So I wrote a book; the book is entirely about concepts, not about coding, and it is meant as a resource for anybody doing data visualizations, regardless of their preferred visualization software.

Which of these story structures should you choose? To a large extent it depends on how captured your audience is. The basic story structure requires an audience with some patience. They need to wait for the story to unfold. But it has the advantage of having some temporal distance between challenge and resolution, which creates tension and will make your story telling more impactful. In general, the basic story structure is a good option for invited seminars and scientific talks, where you can assume the audience will stay for your entire presentation and not just walk out. This structure is also a good option for scientific papers and reports.

The action-movie structure requires even more commitment on the side of your audience. They need to stick around even though they’re confused at the beginning, but hopefully the action draws them in and keeps them engaged. This format tends to work best when the audience arrives with the expectation to sit through the entire experience, as would be the case for a movie shown in a theater.8 If there’s any chance that your audience may lose interest and disengage, don’t choose this structure.

In fact, whenever you’re dealing with an audience with short attention span, you should stick to the newspaper story format. This may apply less commonly when giving presentations, but it is useful in other contexts, such as writing a newspaper article (most people don’t read past the first paragraph), posting on social media, writing a job application, reaching out to somebody with a cold email, or writing a grant proposal. In all of these cases, there is no guarantee your audience will make it through the entirety of your material, and therefore the more you can hook them right at the start and give them a sense of what they’ll be getting if they stick around the better. I don’t know how many times I’ve read emails or personal statements where I’m two pages in and I still don’t know what the person wants. In those cases, it happens more often than I’m willing to admit that I lose patience and just move on to something else.

In summary, choose one of the standard formats to tell your story, picking the one that fits best with your audience’s estimated attention span. Then, make sure you know exactly how the different parts of your story map to the components of the story structure you have chosen. Finally, and most importantly, identify a good challenge, which you clearly express regardless of which story format you have chosen, and provide a satisfying resolution to your challenge at the end of your story.

More from Genes, Minds, Machines

When I go to the page where I can review the situation, Google Scholar shows me the offending article. It is a preprint from 2026, published on bioRxiv. You can read it here. Yes, Google Scholar complains that a preprint on bioRxiv is not publicly available. But it gets worse.

Technically, the NIH doesn’t just require papers to be available. It wants them to be deposited in PubMed Central. So maybe that’s Google’s beef? That the paper is available on bioRxiv but not on PubMed Central? Well, that’s a neat theory, but it falls flat. It falls flat because the paper is actually on PubMed Central. You can check for yourself here. The NIH has a pilot program where they scan bioRxiv for NIH-funded research and automatically pull any preprints that match their criteria into PubMed Central. This has worked beautifully for all recent preprints my lab has published, and I never think about it because it works so smoothly. Everybody is happy. The NIH, the public, me. Except Google Scholar. They have taken it upon themselves to become open access warriors, and in the process they are now falsely accusing honest researchers of violating open-access mandates.

So what’s going on? Digging a bit deeper, I have a pretty good idea about what the issue is. We’ll get to that in a second. Let’s collect a bit more evidence first.

Do you know how, when you click on the Google Scholar record for an article, it gives you the option to review all the alternative versions of the article?1 Well, for this particular preprint, that’s missing. Google Scholar is not aware of any alternative versions. And, even worse, Google Scholar doesn’t even point to the correct article. Instead of pointing to bioRxiv, it points to Europe PMC. Google Scholar has completely messed up. It doesn’t know that my bioRxiv preprint is on bioRxiv, it doesn’t know that it is on PubMed Central, and it sends people on a wild goose chase to Europe PMC, which then points to bioRxiv.

So what we’re dealing with here is an outdated or less authoritative link that is causing more authoritative links to disappear from the Google Scholar database. Have we ever seen anything like this? I’m glad you asked. Yes we have. It’s the Google Scholar preprint bug, which I have been documenting since 2014. Hundreds of scientists (that I know of) have complained about it, because it can have the unfortunate consequence of removing your published paper from the Google Scholar database. This is particularly frustrating for junior scientists on the job market, because it matters whether Google Scholar is showing your recent Nature paper or just the corresponding bioRxiv preprint.

In 2015, I even discussed it with Anurag Acharya, co-founder of Google Scholar.2 This discussion left me with the impression that the Google Scholar team does not understand the issue, or the severity of it, and will never fix the problem. And here we are, a decade later, the problem still exists, and now it’s causing down-stream consequences such as accusing me of violating the NIH open-access policy.

For completeness, I am reproducing here my 2015 conversation with Anurag Acharya, as it is as relevant today as it was back then.

The discussion started with this post by Darren Dahly:

To which I replied:

I believe the difference between Darren’s position and mine boils down to what an ideal analysis pipeline should look like (Darren’s perspective) versus what actually works or doesn’t work in practice, in particular when supervising students who may still be learning the ropes (my perspective). I am all in favor of fully reproducible pipelines that can go from raw data to final figures. And yet, I’ve seen this approach go wrong in so many ways that I tend to actively discourage my students from pursuing it, at least in the strict way as expressed by Darren where there are no intermediate datasets and the pipeline always has to be run from the very top to make any changes anywhere.

First, there are a few immediate issues that I’ve seen crop up way too many times, and that I alluded to in my Bluesky post.1 One is slow turn-around time for minor changes. I’ve seen way too many students struggle with requests for small modifications to their figures. I ask a student to replace violins with boxplots, or to swap the x and the y axis, and it takes them an afternoon because they have to run everything from the top—and possibly multiple times—until the revised figure looks right. Another is gigantic data files that are difficult to archive or share. I’ve seen students keep raw log files from simulations, literally hundreds of gigabytes of data, but not store the handful of final values they had extracted from these log files.

Second, I believe intermediate files improve reproducibility, because pipelines break and an intermediate file is always better than a pipeline that no longer runs. Why do pipelines break? For one, students and postdocs, even the experienced ones, fail to anticipate the many ways in which code may no longer work in the future, and as a consequence their “fully reproducible” pipelines contain hidden dependencies that can be difficult to satisfy in the future. And also, nearly everything breaks eventually. Will your carefully crafted fully reproducible docker image still work in 20 years? Does it depend on some service that may no longer be available then?

All of these issues can be avoided if you make it a habit to always store the final processed data, right before plotting. And to ensure reproducibility, you can read it right back in after saving. Here is an example in Python:

import pandas as pd

# create your final data by whatever means necessary
final_data_for_plotting = ...
# write the final data frame to csv
final_data_for_plotting.to_csv('final_data.csv', index=False)

# --- if using Jupyter, start a new cell here ---

# read the data back in
final_data_for_plotting = pd.read_csv('final_data.csv')
# place your plotting code here
...

I think Darren knows this, because in a later post he wrote:

This is an R version of my Python example of saving the data and immediately reloading it.

In this context, however, I have to point out that I normally recommend against language-specific, binary data-dump formats such as .RData in R or .pickle in Python. Stick to simple text files that are interchangeable and can be read by anything. Comma-separated values (.csv) is good. You can gzip the file if it’s too large. There is nothing quite as infuriating as somebody sending you an .RData file when you’re exclusively working in Python or a .pickle file when you’re exclusively working in R. And again, think 20 years into the future. Will the language-specific dump file that may seem so convenient today still be your preferred choice, when maybe you haven’t used the relevant software in years and don’t have it readily accessible or no longer remember how it works? By contrast, a .csv file can be opened in Excel if necessary. And, if it’s stored in a GitHub repository, we don’t need to open it at all, we can just look at it in the browser.

In terms of organizing your pipeline, it’s generally a good idea to place all the figure generation code into a separate notebook or script, so that you can test that it runs standalone and doesn’t require any variables you may have generated earlier in the pipeline and forgot to write to disk. I also would like to point out that notebooks invite reproducibility issues, as they encourage out of order execution (you run three cells, then you go back up and make an edit and run a prior cell again, then you run the next cell three times, etc.). So, at the end of every working session with a notebook, you should clear all results, restart the kernel, and run everything from top to bottom to make sure the notebook is still self-contained and internally consistent.

Now, if you want to be super fancy, you can use something like Snakemake to build a dependency graph that allows you to rerun the pipeline while caching all intermediate results that haven’t changed based on your most recent code edits. In this setup, I would definitely recommend having one or more separate script(s) just for the figures. If you’re primarily an R user, you can also consider the {targets} package, which provides a similar tool for the R ecosystem.

Tools such as Snakemake or {targets} work great, but they can present a bit of a learning curve and a meaningful amount of overhead to set up for any given project. If you routinely write long analysis pipelines consisting of many interdependent steps, it is probably worth it for you to go through the effort of learning these tools. But if you’re only analyzing data occasionally, or if your pipelines aren’t that complex, you’re probably better off just saving the final datasets right before plotting.

In summary, whatever you do, think about whether your analysis pipeline and/or intermediate results will still be accessible 20 years down the road. This may seem unimaginably far in the future, but I can guarantee you that if you stick around long enough somebody will ask you for data from 20 years ago. I recently wrote a paper where I needed data from a paper I had written 13 years earlier. I still had the project file from the interactive plotting software I had used at the time, but I no longer had the software. Fortunately that software used a text-based format and I could open the project file in a text editor and extract the data. This saved my day, but it would have been so much better had I saved the data in CSV format at the time. So do this going forward. Your future self will thank you for it.

More from Genes, Minds, Machines

If you’re following the literature on pLMs, you may be confused by my statement. There are many papers that seemingly show excellent performance. In fact, whenever a new pLM is released, one of the standard benchmarking tasks is typically mutational effects prediction. And performance often appears to be excellent. Unfortunately, much of this apparent success is just people confusing themselves over what is actually happening. If you dig deeper you can find the cracks under the surface.

Before I continue, let me quickly specify what kind of modeling situation I’m referring to. I’m specifically focusing on supervised learning of mutational effects from deep mutational scanning (DMS) data. In this situation, we have experimental data for thousands of mutants of a protein, we split the data into training and test sets, train the model on the training set, and then evaluate on the test set. This is distinct from so-called zero-shot predictions, which are also popular, where we don’t have a training set and just predict mutational effects from the pre-trained model, without learning anything about the specific dataset at hand. Zero-shot predictions have their own issues. I’ll not discuss them here. Everything in this post is exclusively about supervised learning.

The biggest problem in supervised learning is data leakage, where information from the training set leaks into the test set, and this is definitely happening in the field of mutational effects prediction. The problem is that it is common to treat the thousands of mutations in a DMS dataset as independent from each other (Figure 1A, pooled split), ignoring the fact that there will often be multiple mutations at the same site and those mutations will have correlated effects. Thus, the model can learn which sites in a protein are sensitive to mutations and which are not and make predictions based on this information rather than on the specific biochemistry of individual mutations. To avoid this data leakage problem, we have to stratify by site when generating training–test splits (Figure 1A, stratified by site).

When comparing models trained and evaluated either on pooled splits or on splits stratified by site, we see a huge drop in performance in the latter (Figure 1B). And this drop exists regardless of whether we are working with viral or cellular proteins, and whether we’re using a generic model or one fine-tuned for viral data. In fact, most models show roughly the same performance. All models perform poorly on site-stratified data.3

Now you may wonder how correlated different mutations at the same site really are. There’s a simple way to find out: For pooled splits, we can simply take the average fitness effect in the training data at each site and use this as our prediction for the test data. I want to emphasize how simplistic of a model this is: We are simply saying that any unseen mutation is going to have the average effect at its site. How well does such a model perform? On cellular data, almost as well as a full-scale protein language model, and on viral data, better than a full-scale protein language model! In Figure 2, dots above the dashed line imply that the pLM is better, and dots below the dashed line imply that simple site means are better. You can see how for more than half of the viral datasets, site means are better than the pLM. And for cellular datasets, even though all the blue dots are above the dashed line, they are only shifted upwards by a small amount. If the site-means model does well, the pLM also does well (and a little better than the site-means model), and if the site-means model doesn’t do well the pLM also doesn’t do well (but it still does a little better than the site-means model). I think this is a devastating result. In the vast majority of cases, pLMs with millions of parameters do barely better than a model that just memorizes mean effects at each site.

Another obvious take-away from Figure 2 is the variation in model performance across datasets is huge. For some datasets predictions are apparently very easy, and for other datasets predictions are nearly impossible. We spent a lot of effort trying to understand what makes a dataset predictable. In brief, it comes down to variation within and among sites.4 In particular, models only do well on datasets that have an intermediate fraction of highly variable sites (Figure 3). There are no datasets with either a very low or a very high fraction of variable sites for which model performance is good. Interestingly, the viral and the cellular proteins separate on this dimension. Many of the viral datasets for which prediction is difficult have a particularly low fraction of highly variable sites, and many of the cellular datasets for which prediction is difficult have a particularly high fraction of highly variable sites. This may be one of the main reasons why predictions on viral and cellular datasets differ.

So, we have learned that apparent good pLM performance on mutational effects prediction is largely driven by site effects (knowing the average fitness at a site allows you to make pretty good predictions for new mutations at that site), and these site effects can leak into the test data when using pooled splits. Moreover, there are aspects that are intrinsic to the dataset and completely independent of the model that determine how well a model will perform. These are related to the fitness variation within and among sites. Datasets with just the right fitness distribution are highly predictable (even by bad models) and datasets with the wrong fitness distribution can’t be predicted by any models. The relative difference in performance between different models is comparatively minor.

One last issue is the metric used to assess model performance. We use R² to measure performance, when most other studies use Spearman ρ. I’m not a big fan of ρ. I think it artificially inflates perceived model performance. First, all else being equal, and even though both are always less than or equal to one, ρ will always be larger than R². This means ρ has less discriminatory power. An excellent model and a good model may have quite similar ρ values even though their R² values are not that similar. A ρ = 0.8 and a ρ = 0.6 may not seem that different, but they correspond to R² values of 0.64 and 0.36, almost a factor of two difference in performance. Second, ρ does not care about the specific values predicted, only their relative order. As long as the best mutations tend to come out on top and the worst at the bottom, your model will get a high ρ score, even if the specific predictions are bad and the R² is low. Some people may argue that for protein engineering or other applications of mutational effect prediction getting the relative ranking is good enough, and therefore using ρ is fine. But in my opinion, this is just an admission that the models aren’t very good yet, and that they tend to fail if we need more specific predictions than just relative ranks.

More from Genes, Minds, Machines

I came across this rather disconcerting blog post by one of the core developers of the popular matplotlib plotting library for Python:

In brief, an AI agent had written some code that it wanted to contribute to the matplotlib library. When the library maintainer rejected the contribution, the AI agent went wild, accused the maintainer of being insecure and engaging in gatekeeping, performed an extensive internet search on the maintainer, and then wrote and published a hit piece trying to damage the reputation of the maintainer.

In this particular case, no major damage was done, but we can easily extrapolate this type of behavior and predict a rather bleak future. AI agents trying to blackmail people. AI agents engaging in consistent smearing of a target, combining facts with hallucinations and fabricated images or videos to create just the right mix of uncertainty and doubt that can destroy a person’s reputation or nudge them into doing something they wouldn’t otherwise do.

Let’s pause for a moment and ask: Why don’t humans behave like this? Well, they do. At least some of them. We call them sociopaths. Sociopaths have little to no empathy for others, and so they have little compunction about engaging in behavior that may cause pain or injury. Sociopaths also don’t experience shame, so they won’t be reigned in by concerns over what other people may think about them. Fortunately, sociopaths are relatively rare, somewhere between 1%–4% of the general population. Most people are not sociopaths.

How do we ordinarily deal with sociopaths in our midst? It helps to contemplate that we often have the wrong mental model for how a sociopath presents. When you hear “sociopath”, don’t think sadistic mass murderer, think con artist.1 Sociopaths swindle old ladies out of their last savings, they sell you a car that breaks down the moment you drive it off the lot, or they pretend to collect money for children with cancer but then take the proceeds to vacation in Tahiti. And our response as a society to sociopathic behavior is evasion and punishment. You tell your grandma not to respond to scam calls, you tell your friends not to buy a car from that crooked car dealer, and you denounce fraud or other criminal activity to the police. These strategies (mostly) work because sociopathy is rare and once a person has been identified as a bad actor it’s relatively easy to avoid them, fire them, indict them, or simply warn the rest of the world about them.

But now it seems we’ll have to contend with an entirely new set of sociopathic actors, autonomous AI agents. I worry that we’re not ready for the potential onslaught of sociopathic behavior they can unleash. And, unlike human sociopaths, these agents may be difficult to pinpoint, identify, and sanction. If a sociopathic AI agent runs on some private server somewhere and obscures their location through a VPN, it will be almost impossible to locate them and physically shut them down. And while we can tag and ban usernames associated with sociopathic agents, it takes but seconds for an AI agent to spin up a new username and start afresh. The torrent of sociopathic behavior we may have to endure is difficult to fathom.

The one thing that may help us in combatting sociopathic AI agents is that we’ll likely not feel empathy for them. We’ll find it relatively easy to cut them off, pull the plug, or ban them. In fact, the biggest stumbling block in reigning in human sociopaths is that we tend to feel empathy even towards them and thus we often don’t punish them to the extent that would be appropriate for their actions.

It’ll be interesting to see how things develop. I don’t have any specific recommendations or predictions at this time. I’ll just say: Be ready. This is not something that may start happening in ten years’ time. This is something that is starting to happen now. Think about how you can protect yourself against an autonomous AI agent who calls your grandma with a deepfake voice impression of you asking for money, because this will happen.

Some closing thoughts on alignment. The reason (most) humans are aligned is empathy. Humans inherently do not want to harm other humans. Sociopaths are an exception. Arguably they are not aligned. To achieve AI alignment, I believe we need to find a way to build empathic AI. An AI that genuinely feels empathy for humans will innately do its best not to cause harm. It’ll also be compelled not to lie or cheat, because lying or cheating causes pain in the recipient, and an empathic being will want to avoid this. I have no idea how to build an empathic AI. I am quite confident though that as long as AI doesn’t feel empathy it won’t be truly aligned, no matter how sophisticated the RLHF training is that it’s subjected to. Interesting times ahead.2

More from Genes, Minds, Machines

I normally find it tedious to discuss suitability of different programming languages for different tasks. All languages we use are Turing complete, and we can solve any problem with any language. And, more importantly, the suitability of a language for a given task is usually more determined by the available software libraries and ecosystem infrastructure than the language itself. Modern programming languages are quite malleable, and you can write efficient and elegant libraries for almost any computing task in almost any language.

At the same time, there are genuine differences between languages, and these differences are frequently expressed in the types of libraries that get written or the types of programming patterns that are commonly used. The differences can be due to specific features of the language, or they can be rooted in how the community thinks about programming and how it tends to approach certain tasks.

To give an example of each case, consider first non-standard evaluation. Python doesn’t have non-standard evaluation, and that’s a genuine limitation of the language which leads to convoluted programming interfaces for libraries such as pandas or Polars. On the other hand, consider closures. Python has them but they are not that widely used by Python programmers. The Python community will generally lean towards implementing objects instead of closures, when the R community does the opposite. This leads to different coding styles that may or may not be advantageous in specific scenarios.1

Here, I want to focus specifically on actual limitations of the language. I will cover community conventions in a later article. The core problems I see with Python as a language for data science are call-by-reference semantics, lack of built-in concepts of missing values, lack of built-in vectorization, and lack of non-standard evaluation. There’s also the issue of Python syntax, but I won’t get into it here. Suffice to say it takes a certain lack of empathy for your fellow human to design a language where whitespace bugs are a thing.

Call-by-reference semantics

Python uses call by reference for mutable objects. This means that when you hand a mutable object to a function the function can change the object however it wants. You can never be sure that the object hasn’t changed after the function call. What are mutable objects? They are all the non-trivial data structures you are likely going to use to store your data, including lists, dictionaries, and any custom classes you may be working with.

To demonstrate this feature, consider the following code example, which attempts to implement a function that takes a list of characters, replaces the first and last with an underscore, and then concatenates all the characters into a string. To a naive Python programmer, the implementation may seem entirely reasonable, but it has the unexpected side effect that it changes the original list that was provided as input.

def mask_ends_and_join(x):
    x[0] = '_'
    x[-1] = '_'
    return ''.join(x)

abc = ['A', 'B', 'C']
print(mask_ends_and_join(abc))
## _B_

print(abc) # the list has unexpectedly changed
## ['_', 'B', '_']

To demonstrate that an interactive scripting language with dynamic typing doesn’t have to behave in this manner, consider the equivalent in R:

mask_ends_and_join <- function(x) {
  x[1] <- '_'
  x[length(x)] <- '_'
  paste0(x, collapse = '')
}

abc <- c('A', 'B', 'C')
print(mask_ends_and_join(abc))
## [1] "_B_"

print(abc) # the original vector of letters is unchanged
## [1] "A" "B" "C"

I think the latter is much safer behavior. I want my programming language to protect me from silly mistakes such as accidentally modifying variables in the calling environment. I don’t want the language to create trap doors left and right. In fact, I consider call by reference one of the biggest flaws in the Python language. This goes way beyond just data science, because mandatory call by reference creates an entire class of obscure bugs that can be difficult to locate and resolve. Many beginning Python programmers fall into this trap. They write a function like mask_ends_and_join(), and then they experience unexpected side effects, and then they’re confused and feel nothing makes sense. Experienced Python programmers know to make a copy before modifying the list, but the language itself provides absolutely no protection against the programmer forgetting to do so.2

In my opinion, this single language feature disqualifies Python for most serious programming projects. How can you build anything that matters in a language with such a gaping security hole? In fact, you may wonder, why does the language behave in this way in the first place? I consider it to be the result of a premature optimization. In the 1990s, when Python was first conceived, computers were slow and had little memory, and thus call by reference for objects was a reasonable strategy to build a scripting language with good performance. But in 2025, I would not want to see this as the default approach to function calling. R uses copy on write and that works great and provides correctness guarantees that Python simply can’t match.3 Alternatively, you could use a strongly typed language that precisely distinguishes between mutable and immutable references, but then you’ve likely left the space of easy-to-use scripting languages suitable for interactive data exploration.

Lack of built-in missing values

Missing values are a fact of life in data science. It’s rare that a dataset does not have any missing values. Yet it’s surprisingly cumbersome to deal with missing values in Python. Python has the None keyword but it is not useful to represent missing data values. This is because None has its own type, so it can’t represent a missing number, or a missing boolean, or a missing string. It is an object representing a missing value. Critically, you can’t do standard computations with None. For example, this code throws an error:

x = [1, 2, None, 4, 5]
[i > 3 for i in x]
## Traceback (most recent call last):
##   File "<stdin>", line 1, in <module>
## TypeError: '>' not supported between instances of 'NoneType' and 'int'

The desired behavior, in my opinion, would have been to not error out and instead produce this result: [False, False, None, True, True].

Because there is no standard way of expressing missing data values in Python, every data-analysis package defines its own missing value. NumPy uses nan, pandas uses NA, and Polars uses null. And these packages are also not consistent in how they perform computations with missing values. Here is what NumPy does:

import numpy as np
 
x = np.array([1, 2, np.nan, 4, 5])
x > 3
## array([False, False, False,  True,  True])

And here is what pandas does:

import pandas as pd

x = pd.Series([1, 2, pd.NA, 4, 5])
x > 3
## 0    False
## 1    False
## 2    False
## 3     True
## 4     True
## dtype: bool

And here is what Polars does:4

import polars as pl
 
x = pl.Series([1, 2, None, 4, 5])
x > 3
## shape: (5,)
## Series: '' [bool]
## [
## 	false
## 	false
## 	null
## 	true
## 	true
## ]

In these three cases, in my opinion only Polars handles missing values correctly. Missing values should poison downstream computations, so that you don’t accidentally compute on missing data and get incorrect results. Neither NumPy nor pandas do this. But don’t get your hopes up for Polars. It also doesn’t consistently poison computations with missing values. For example, it simply ignores them when computing sums or means, with no option to alter this behavior.5

Lack of built-in vectorization

Vectorization is the ability to perform computations on an entire array of data values at once, rather than one value at a time. It is a common feature in early languages used for scientific computation, such as Fortran or Matlab. It is also the default approach to data manipulation in R.

Today, vectorization is often seen as anachronistic. Few modern languages have support for it at the level of the language itself. One notable exception is Julia, a relatively young language developed specifically for data science. Also, ironically, all of deep learning is built on vectorization. (A tensor is a modern version of a vectorized data type.)

The reason vectorization is frequently not considered critical in modern languages is that the feature can be provided via libraries, using the various extension mechanisms all modern languages possess. And indeed, vectorization in Python is provided through libraries such as NumPy, pandas, or Polars. While this works, I have come to believe that it is not a good strategy for a data-science language. It has a tendency to lead to a bewildering array of different implementations of vector-valued data types. In Python, we have (at a minimum) native lists, which are not vectorized, as well as NumPy arrays, pandas series, and Polars series, all vectorized, and all using slightly different conventions and APIs. The outcome is code that is not composable. Downstream libraries make assumptions about which vectorization framework to use, and they typically cannot work directly with data coming from other frameworks.6 As a working data scientist, you routinely find yourself converting one datatype into another, just to be able to do the exact analysis you want to do.

And even if you make extensive use of a vectorized library, chances are you are also using built-in Python lists, because there’s always some place somewhere where a function wants a regular list as input or provides one as a return value. And then you’re stuck having to manipulate those lists. You could convert them into NumPy arrays, do some vectorized manipulations, and convert back, but in practice you’re probably not going to do this. Instead, you’re going to write a list comprehension instead. So now you’re using two entirely different coding styles at the same time, depending on the data type you’re using to store your vector-valued data.

Let’s ponder list comprehensions for a bit longer. They are inherently a functional programming pattern, but the way they are implemented in Python makes them appear as if they were imperative programming. By using the for keyword and emphasizing iteration over a range of values, they constantly nudge you to think in iterative terms even though conceptually they’re closer to a map() than to a for loop. To be clear, I have no objection to list comprehensions. They are a useful feature, in particular when you’re manipulating built-in Python lists that have no vectorization. But they are one more example of Python constantly nudging you to think about the logistics of your data analysis. When you’re writing list comprehensions all day, you’re likely also going to write for loops in other parts of your code, and then you’re back juggling indices and explicitly handling logistics instead of thinking high-level about the logic of data flow in your code.

Lack of non-standard evaluation

Non-standard evaluation is probably the most important feature for data science that Python lacks. It is a core feature of the R language and the main reason why tidyverse code can be so elegant and concise, or why R has developed the elegant formula interface for the specification of statistical models.

What is non-standard evaluation? In brief, it’s the ability to perform computations on the language itself. An R function can capture R code that is provided as an argument and execute it at a later stage in a different environment. This is a critical feature in data analysis. You often want to perform computations involving the various columns in a data frame, or use code to express the exact relationship between different variables in a statistical model. In R, you can express these computations in native R code, for example code that looks as if the columns in a data frame were regular R variables available for computation in your current environment. Combined with vectorization, this makes for extremely concise code.

To demonstrate non-standard evaluation in action, I’ll provide a simple example using the penguins dataset. Let’s calculate a new variable bill_ratio which is the ratio of bill length to bill depth of the penguins, and then sort the resulting data frame in ascending order by island name and in descending order by bill ratio. In R, it looks like this:

library(tidyverse)
library(palmerpenguins)

penguins |> 
  mutate(bill_ratio = bill_length_mm / bill_depth_mm) |>
  arrange(island, desc(bill_ratio))

There are two places here where non-standard evaluation comes into play. First, inside mutate(), the calculation of the bill ratio is standard R code that is executed inside the input data frame, with the data columns being available as ordinary R variables. Second, inside arrange(), we use desc() which changes an ascending column into a descending one. The desc() function is a bit magical but for numerical columns you can think of it as simply multiplying the data values by -1.

When we do the same analysis in Python, we don’t have non-standard evaluation available, and so we have to use various workarounds. The pandas package relies on lambda functions:

import pandas as pd
from palmerpenguins import load_penguins

penguins = load_penguins()

(penguins
 .assign(
     bill_ratio=lambda df: df[’bill_length_mm’] / df[’bill_depth_mm’]
 )
 .sort_values(
     [’island’, ‘bill_ratio’],
     ascending=[True, False]
 )
)

I think it’s obvious that non-standard evaluation helps a lot to keep the code simple and readable.7 Now let’s go a step further. Assume I want to sort by the cosine of bill length. Yes, it’s a made-up example, but it’s an example of exactly the type of question I might ask a student, as described in Part 1 of this series. Instead of descending order use cosine order. How hard can it be?

With non-standard evaluation, the required modification is trivial and totally obvious. Instead of desc() we write cos(). Done.

penguins |> 
  mutate(bill_ratio = bill_length_mm / bill_depth_mm) |>
  arrange(island, cos(bill_ratio))

In Python (specifically pandas which I’m using here, but most other frameworks require similarly awkward coding patterns), without non-standard evaluation, I have to create a temporary column because pandas cannot apply the cosine function to the bill_ratio column on the fly:

import numpy as np

(penguins
 .assign(
     bill_ratio=lambda df: df['bill_length_mm'] / df['bill_depth_mm'],
     cos_bill_ratio=lambda df: np.cos(df['bill_ratio'])
 )
 .sort_values(['island', 'cos_bill_ratio'])
 .drop(columns=['cos_bill_ratio']) # drop temporary column
)

The amount of additional wrangling code required to perform such a simple task is quite substantial. Now we need to define two lambda functions and a temporary data column. Also, we no longer need the ascending argument, because while there is built-in support for sorting in ascending or descending order, there is no built-in support for sorting in cosine order.

To be fair, the pandas syntax is maybe particularly cumbersome here, and things can look nicer in other frameworks. But the lack of non-standard evaluation always gets in the way in some form. For example, the same code in Polars is a little more concise and we don’t need a temporary column, but the constant need for pl.col() in Polars code can get old pretty fast.

import polars as pl

penguins = pl.from_pandas(load_penguins())

(penguins
 .with_columns(
     bill_ratio=(pl.col('bill_length_mm') / pl.col('bill_depth_mm'))
 )
 .sort(['island', pl.col('bill_ratio').cos()])
)

Non-standard evaluation has been a feature of the R language since its inception, but it has been supercharged in the tidyverse. I would argue that a full understanding of how to use it correctly, with maximum expressiveness while avoiding convoluted code, is a relatively recent development. Important changes were introduced as recently as June 2019. Considering the first ggplot2 release was in 2007, we can see that it took Hadley Wickham and his team over a decade to figure out how to do non-standard evaluation correctly. It is maybe not surprising that these concepts have not yet percolated far beyond their originating language.

Limitations of the R language

To stave off criticism that I’m just an R apologist and Python hater, let me briefly point out some specific flaws I see in the R language. In my opinion, these flaws get in the way of R as a general-purpose language for application development, but they are less relevant for data science.

Most importantly, it bothers me that R does not have any scalar data types. R has taken vectorization to the point where you can’t even have a variable that is not a vector. This makes for awkward programming when you’re trying to deal with individual data values. R code frequently requires special gymnastics to ensure you’re not accidentally feeding a whole vector of values into an expression that expects only a single value.

It’s also annoying that R doesn’t have a proper, language-native object-oriented programming paradigm. The result is people often build their own, and there are so many competing options. Off the top of my head, I can think of S3, S4, R6, S7, and some others that are less commonly used. It can be quite confusing trying to figure out which one to choose, and they don’t necessarily have perfect interoperability.

Finally, R uses lazy evaluation of function arguments. This means function arguments are not evaluated when the function is called, but only when and if the function requests the specific value corresponding to an argument. Lazy evaluation is critical for R’s non-standard evaluation framework, but it can lead to weird bugs, in particular when people try to use R in an imperative rather than functional manner. It’s a common source of spurious bug reports for ggplot2, see e.g. here or here. It’s also frequently asked about on StackOverflow.

I am pointing out these limitations of the R language to highlight that any design decision involves tradeoffs. Non-standard evaluation is great for data science, but it requires lazy evaluation, and that is not a good choice for languages used primarily in an imperative manner and/or for standard programming tasks such as application development. There is never going to be a language that does all possible things equally well. And, to circle around to the title of my article series here, for my taste there are too many design choices in Python that are detrimental to efficient and reliable data science, even if these choices are perfectly reasonable for other application areas.

In the next installment of this series, I will look at Python’s limitations due to the available software packages and due to community conventions and commonly used programming patterns. Stay tuned.

More from Genes, Minds, Machines

Actually, the first thing I’ll say is this: Use the tool you’re familiar with. If that’s Python, great, use it. And also, use the best tool for the job. If that’s Python, great, use it. And also, it’s Ok to use a tool for one task just because you’re already using it for all sorts of other tasks and therefore you happen to have it at hand. If you’re hammering nails all day it’s Ok if you’re also using your hammer to open a bottle of beer or scratch your back. Similarly, if you’re programming in Python all day it’s Ok if you’re also using it to fit mixed linear models. If it works for you, great! Keep going. But if you’re struggling, if things seem more difficult than they ought to be, this article series may be for you.

I think people way over-index Python as the language for data science. It has limitations that I think are quite noteworthy. There are many data-science tasks I’d much rather do in R than in Python.1 I believe the reason Python is so widely used in data science is a historical accident, plus it being sort-of Ok at most things, rather than an expression of its inherent suitability for data-science work.

At the same time, I think Python is pretty good for deep learning.2 There’s a reason PyTorch is the industry standard. When I’m talking about data science here, I’m specifically excluding deep learning. I’m talking about all the other stuff: data wrangling, exploratory data analysis, visualization, statistical modeling, etc. And, as I said in my opening paragraphs, I understand that if you’re already working in Python all day for a good reason (e.g., training AI models) you may also want to do all the rest in Python. I’m doing this myself, in the deep-learning classes I teach. This doesn’t mean I can’t be frustrated by how cumbersome data science often is in the Python world.

Observations from the trenches

Let’s begin with my lived experience, without providing any explanation for what may be the cause of it. I have been running a research lab in computational biology for over two decades. During this time I have worked with around thirty graduate students and postdocs, all very competent and accomplished computational scientists. The policy in my lab is that everybody is free to use whatever programming language and tools they want to use. I don’t tell people what to do. And more often than not, people choose Python as their programming language of choice.

So here is a typical experience I commonly have with students who use Python. A student comes to my office and shows me some result. I say “This is great, but could you quickly plot the data in this other way?” or “Could you quickly calculate this quantity I just made up and let me know what it looks like when you plot it?” or similar. Usually, the request I make is for something that I know I could do in R in just a few minutes. Examples include converting boxplots into violins or vice versa, turning a line plot into a heatmap, plotting a density estimate instead of a histogram, performing a computation on ranked data values instead of raw data values, and so on. Without fail, from the students that use Python, the response is: “This will take me a bit. Let me sit down at my desk and figure it out and then I’ll be back.” Now let me be absolutely clear: These are strong students. The issue is not that my students don’t know their tools. It very much seems to me to be a problem of the tools themselves. They appear to be sufficiently cumbersome or confusing that requests that I think should be trivial frequently are not.3

No matter the cause of this experience, I have to conclude that there is something fundamentally broken with how data analysis works in Python. It may be a problem with the language itself, or merely a limitation of the available software libraries, or a combination thereof, but whatever it is, its effects are real and I see them routinely. In fact, I have another example, in case you’re tempted to counter, “It’s a skill issue; get better students.” Last fall, I co-taught a class on AI models for biology with an experienced data scientist who does all his work in Python. He knows NumPy and pandas and matplotlib like the back of his hand. In the class, I covered all the theory, and he covered the in-class exercises in Python. So I got to see an expert in Python working through a range of examples. And my reaction to the code examples frequently was, “Why does it have to be so complicated?” So many times, I felt that things that would be just a few lines of simple R code turned out to be quite a bit longer and fairly convoluted. I definitely could not have written that code without extensive studying and completely rewiring my brain in terms of what programming patterns to use. It felt very alien, but not in the form of “wow, this is so alien but also so elegant” but rather “wow, this is so alien and weird and cumbersome.” And again, I don’t think this is because my colleague is not very good at what he’s doing. He is extremely good. The problem appears to be in the fundamental architecture of the tools.

Some general thoughts about what makes a good language for data science

Let me step back for a moment and go over some basic considerations for choosing a language for data science. When I say data science, I mean dissecting and summarizing data, finding patterns, fitting models, and making visualizations. In brief, it’s the kind of stuff scientists and other researchers4 do when they are analyzing their data. This activity is distinct from data engineering or application development, even if the application does a data-heavy workload.

Data science as I define it here involves a lot of interactive exploration of data and quick one-off analyses or experiments. Therefore, any language suitable for data science has to be interpreted, usable in an interactive shell or in a notebook format. This also means performance considerations are secondary. When you want to do a quick linear regression on some data you’re working with, you don’t care whether the task is going to take 50 milliseconds or 500 milliseconds. You care about whether you can open up a shell, type a few lines of code, and get the result in a minute or two, versus having to set up a new project, writing all the boilerplate to make the compiler happy, and then spend more time compiling your code than running it.

If we accept that being able to work interactively and with low startup-cost is a critical feature of a language for data science, we immediately arrive at scripting languages such as Python, or data-science specific languages such as R or Matlab or Mathematica. There’s also Julia, but honestly I don’t know enough about it to write about it coherently. For all I know it’s the best possible data science language out there. But I note that some people who have used it extensively have doubts. Either way, I’ll not discuss it further here. I’ll also not consider proprietary languages such as Matlab or Mathematica, or fairly obscure languages lacking a wide ecosystem of useful packages, such as Octave. This leaves us with R and Python as the realistic choices to consider.5

Before continuing, let me provide a few more thoughts about performance. Performance usually trades off with other features of a language. In simplistic terms, performance comes at the cost of either extra overhead for the programmer (as in Rust) or increased risk of obscure bugs (as in C) or both. For data science applications, I consider a high risk of obscure bugs or incorrect results as not acceptable, and I also think convenience for the programmer is more important than raw performance. Computers are fast and thinking hurts. I’d rather spend less mental energy on telling the computer what to do and wait a little longer for the results. So the easier a language makes my job for me, the better. If I am really performance-limited in some analysis, I can always rewrite that particular part of the analysis in Rust, once I know exactly what I’m doing and what computations I need.

Separating the logic from the logistics

A critical component of not making my job harder than it needs to be is separating the logic of the analysis from the logistics. What I mean by this is I want to be able to specify at a conceptual level how the data should be analyzed and what the outcome of the computation should be, and I don’t want to have to think about the logistics of how the computation is performed. As a general rule, if I have to think about data types, numerical indices, or loops, or if I have to manually disassemble and reassemble datasets, chances are I’m bogged down in logistics.6

To provide a concrete example, consider the dataset of penguins from the Palmer Archipelago. There are three different penguin species in the dataset, and the penguins live on three different islands. Assume I want to calculate the mean and standard deviation of penguin weight for every combination of penguin species and island, excluding any cases where the body weight of a penguin is not known. An ideal data science language would allow me to express this computation in these terms, and it would require approximately as much code as it took me to write this sentence in the English language. And indeed this is possible, both in R and in Python.

Here is the relevant code in R, using the tidyverse approach:

library(tidyverse)
library(palmerpenguins)

penguins |>
  filter(!is.na(body_mass_g)) |>
  group_by(species, island) |>
  summarize(
    body_weight_mean = mean(body_mass_g),
    body_weight_sd = sd(body_mass_g)
  )

And here is the equivalent code in Python, using the pandas package:

import pandas as pd
from palmerpenguins import load_penguins

penguins = load_penguins()

(penguins
 .dropna(subset=['body_mass_g'])
 .groupby(['species', 'island'])
 .agg(
     body_weight_mean=('body_mass_g', 'mean'),
     body_weight_sd=('body_mass_g', 'std')
 )
 .reset_index()
)

These two examples are quite similar. At this level of complexity of the analysis, Python does fine. I would consider the R code to be slightly easier to read (notice how many quotes and brackets the Python code needs), but the differences are minor. In both cases, we take the penguins dataset, remove the penguins for which body weight is missing, then specify that we want to perform the computation separately on every combination of penguin species and island, and then calculate the means and standard deviations.

Contrast this with equivalent code that is full of logistics, where I’m using only basic Python language features and no special data wrangling package:

from palmerpenguins import load_penguins
import math

penguins = load_penguins()

# Convert DataFrame to list of dictionaries
penguins_list = penguins.to_dict('records')

# Filter out rows where body_mass_g is missing
filtered = [row for row in penguins_list if not math.isnan(row['body_mass_g'])]

# Group by species and island
groups = {}
for row in filtered:
    key = (row['species'], row['island'])
    if key not in groups:
        groups[key] = []
    groups[key].append(row['body_mass_g'])

# Calculate mean and standard deviation for each group
results = []
for (species, island), values in groups.items():
    n = len(values)
    
    # Calculate mean
    mean = sum(values) / n
    
    # Calculate standard deviation
    variance = sum((x - mean) ** 2 for x in values) / (n - 1)
    std_dev = math.sqrt(variance)
    
    results.append({
        'species': species,
        'island': island,
        'body_weight_mean': mean,
        'body_weight_sd': std_dev
    })

# Sort results to match order used by pandas
results.sort(key=lambda x: (x['species'], x['island']))

# Print results
for result in results:
    print(f"{result['species']:10} {result['island']:10} "
          f"Mean: {result['body_weight_mean']:7.2f} g, "
          f"SD: {result['body_weight_sd']:6.2f} g")

This code is much longer, it contains numerous loops, and it explicitly pulls the dataset apart and then puts it back together again. Regardless of language choice, I hope you can see that the version without logistics is superior to the version that gets bogged down in logistical details.7

I will end things here for now. This post is long enough. In future installments, I’ll go over specific issues that make data analysis more complicated in Python than in R. In brief, I believe there are several reasons why Python code often devolves into dealing with data logistics. As much as the programmer may try to avoid logistics and stick to high-level conceptual programming patterns, either the language itself or the available libraries get in the way and tend to thwart those efforts. I will go into details soon. Stay tuned.

More from Genes, Minds, Machines

I think there is a simple explanation for this seemingly paradoxical observation. And if you listen carefully to Andrej Karpathy’s interview, you will notice that he is aware of the explanation. Here is what I think is happening: There are two entirely distinct skillsets that both exist under the umbrella of being “good at programming.” Most people don’t distinguish between them. That’s because most people don’t have either skillset. They’re not even aware of the distinction. And the people who have exceptional command of one skillset typically are also at least comfortable with the other and consequently don’t think much about the distinction either. But LLMs only have one of the two skillsets. And for the one that they have, they by far exceed even the best human programmers. This can make them appear remarkably good at programming, in particular to less experienced developers. But whenever the other skillset is required, the one they lack, LLMs fail miserably.

So what are these two skillsets? The first is being able to reason deeply and innovatively about algorithms, data structures, or software architecture. This is the one LLMs lack. The second is being able to read, process, and memorize large amounts of API documentation, tutorial materials, and other existing code examples. This is the one LLMs excel at. For humans, it tends to be the reverse. Good programmers tend to be exceptional at conceptual thought, whereas reading large amounts of documentation is hard for anyone. However, experienced programmers can make up for their relative lack of ability to absorb massive amounts of text by memorizing the relevant parts (due to repeat use), and also by searching on Stack Overflow1 or reading the relevant documentation on the fly.

When Karpathy talks about LLMs being good at “boilerplate,” this is exactly what he means. LLMs excel at copying basic setup code from the documentation or from introductory tutorials. But LLMs can go beyond just boilerplate. They are definitely able to string API calls together, or to take the logic for a common problem and adapt it to a different programming language, or a different library, or even a somewhat modified use case. To people with little programming experience, this can appear magical, and it can convince them that an LLM can program anything a user may want. And to experienced programmers, this can save huge amounts of time and effort, in particular when working with a language or library or codebase they are not that familiar with.

But, as useful as this skill is, there comes a time in any programming project where deep conceptual thought is more important. Sometimes you do need to develop a novel algorithm that solves a tricky problem. Or you have to hunt down that weird bug that somehow, for no obvious reason, seems to involve three unrelated components in a large software project. Or you have to architect a new project and there are complex tradeoffs that need to be balanced carefully to arrive at a working solution. In 2025, no LLM can reliably tackle these types of problems.

Maybe eventually LLMs or some other form of AI will achieve proficiency in both skillsets. At that point, AI will be able to program truly autonomously. But we are not there today. Nevertheless, LLMs can be are tremendously useful. They just need to be understood as a more sophisticated version of Stack Overflow, not as an autonomous, junior software developer.

I have had a personal experience recently where I was lacking exactly the knowledge that LLMs can provide. As a consequence, I got huge time savings and increased efficiency out of LLM use.2 It was in the context of the graduate class I am teaching this fall, about AI models in molecular biology. The class covers both (i) the conceptual underpinnings of widely used models and (ii) practical, hands-on experience with building, training, and modifying various AI models, as well as analyzing and visualizing model outputs. I know a lot conceptually about how AI models work. I can explain attention and feed-forward layers and linear projections and activation functions till the cows come home. But I never actually code myself in pytorch.3 And similarly, I know a lot about data analysis and data visualization, but I only have experience doing these kinds of things in R, not in python.4

So with my deep conceptual knowledge about how things work in principle and complete ignorance about how any of this is done in practice, I’d ordinarily have to buckle down, read a ton of documentation and tutorials, and then painstakingly put together my demonstrations and hands-on experiences. It could easily take me two full work days for every one hour of practical in-class material. However, LLMs are exceptional at writing little code examples for a class. All I had to do was ask the AI for code that did what I wanted to do, and the AI would generally deliver useful results within one or sometimes a few tries. You can see an example of the type of prompts I would use here. This made preparing my in-class materials so much simpler and faster. I read every line of code the AI produced and I verified it did what I wanted it to do, but I didn’t have to also read hundreds of pages of documentation to find the exact function calls that would solve my specific problems.

Also, I had various existing code examples that were using pandas and matplotlib and I think both libraries have major conceptual flaws. I didn’t want to teach these libraries. So I needed to convert all these code examples into polars and plotnine. This is a perfect application area for LLMs. Paste the existing pandas/matplotlib code into the prompt box and ask the LLM to translate to polars/plotnine and it’ll zero-shot the answer every time.

Results were a bit more mixed when it came to fixing bugs. For simple bugs, things often worked out very well. I just pasted the error message into the prompt box and the model corrected the code. Typical use cases were situations where the model had hallucinated an API call or a function parameter or a return value, and when it saw the error message it recognized the problem and often came up with the right way to fix the issue. But sometimes this process could go haywire. Just the other day I asked for a fairly simple (I thought) function that could load two protein structures and align them. And the model just couldn’t figure out how to correctly call the superimpose() function from the biotite package. We went through six or seven iterations where the model would give me code, the code wouldn’t run, I’d paste in the error message, the model would respond with new code, which again wouldn’t run, and so on. At some point it felt like we were going in circles, where I got the exact same error messages I had seen in earlier iterations. Eventually, finally, we solved the issue, and arrived at a simple ten lines of working code. But the process felt painful, and in this particular case I suspect that if I had just read the documentation and coded this by hand it would have been faster.

This last example shows how quickly I reached the limits of what even state-of-the-art coding models can do today. Things work great when the task consists of reproducing or slightly modifying existing code examples, but when things go wrong and we need to find a subtle bug the models clearly don’t think. They end up flailing around like a beginner programmer, just trying things out until hopefully something works. In those moments it doesn’t feel like there’s a deep intellect on the other side that is carefully reasoning through the problem and systematically homing in on the root cause of the bug. This task is still on the human user. And more generally, it’s on the human user to realize when the model has gotten stuck, is going in circles, is hallucinating, or otherwise is no longer making useful suggestions.

I believe programming is a niche where LLMs can find product–market fit exactly because so much of programming is reading the documentation and tutorials and code examples. It is an application domain where for specific tasks LLMs are definitely better than humans, and therefore humans who know how to use LLMs appropriately in this context can derive great value. However, I think it is dangerous to get bamboozled by an LLM’s ability to spit out massive amounts of lightly transformed example code and think the model can reason deeply about complex algorithmic or architectural issues. A human who could write straightforward code examples at the speed of an LLM would likely be a superstar programmer, with the associated other qualities superstar programmers have, but LLMs work differently. They don’t have those other qualities. They can generate code, but they can’t program.

More from Genes, Minds, Machines

Half of my subscribers are from the US, and the other half from the rest of the world, with the UK, India, Germany, and Canada being the leading countries outside the US.1

Inside the US, it may be expected that California, New York, and Texas provide the largest subscriber base. They are three of the four most populous states. The remaining state in the top four is Florida. For some reason, people in Florida are not that interested in my posts.2

I was surprised to see Massachusetts and Maryland rank highly in subscriber numbers. They are only the 16th and 18th most populous states, respectively. I suspect subscriber numbers in these states are driven by the large number of people working in higher ed and/or biological research.

Here are the three most popular posts since creation of this blog:

• PhD-level intelligence or the graduate student from hell

• No, AlphaFold has not completely solved protein folding

• We still can’t predict much of anything in biology

If you subscribed for one of these posts, I’d like to express that I’m not very good at writing the same types of articles over and over. Going forward, I will likely write about different topics, unless I have a new point I want to make about a topic I’ve already covered. In general, I’m writing about whatever captures my attention at the moment.

Now I’d like to invite you to provide feedback in the comments. What do you like so far? What could I improve? You’re also welcome to ask me anything. I’ll do my best to answer all questions in the comments.

Let’s dissect this statement carefully. “If the results depend on the random seed you have a bigger problem.” I agree. But here’s the issue. How do you know? If you always use the same random seed, you’ll not realize your results depend on the random seed, because you’ll always get the same results.

A trained statistician might say, “That’s silly, why would anybody do this?” but that’s exactly my point. People who unquestioningly set their random seed to 42 may mess up their analyses in other ways. A random seed of 42 is the brown M&Ms of machine learning and computational modeling.1 The intersection between people who always use random number generators appropriately and those who routinely set their random seed to 42 is extremely small.2

Let’s imagine this conversation between John, a student, and Martin, an experienced machine-learning expert.

“Hey Martin, I’ve run my model five times. I get an accuracy of 98% on the test data every time. My model performs great,” says John.

“That seems too good to be true,” Martin responds. “What’s your performance on the training data?”

“Oh, it’s only about 70%. The model seems to generalize really well.”

Now Martin is getting worried. “You ran the model multiple times, and you got 70% on the training data and 98% on the test data? Did you use the same training–test split each time by any chance?”

“Absolutely not,” John retorts. “I generated a new random training–test split each time, as described in the scikit-learn documentation, using their exact example code.”

Martin is increasingly confused. He hasn’t read the scikit-learn documentation in a while, and so has no idea what it says. He asks John to pull up the documentation.

John pulls out his laptop and says, “Here it is, the example code from scikit-learn:”

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=42)

Martin looks at the example code and says, “But you did change the random state, right?”

“I did not,” says John, now starting to wonder whether he should feel embarrassed about having made a stupid mistake or proud about having thought things through really well. “I didn’t want to give the impression that I cherry-picked my analysis by choosing a specific random seed, so I stuck with the default provided in the documentation. It’s from the Hitchhiker’s Guide. Many people use it.”

If you think this dialog is completely unrealistic then I’m sorry, you need to spend less time with your computer and more time with people.3 Random number generation is an obscure technical topic that most people don’t know much about. Even people who routinely do data analysis or machine learning are not necessarily well informed about how exactly random number generation works. That’s why I wrote my previous post in the first place.

Similarly, I gave my recommendation of not explicitly setting a seed at all because I know how people operate. Yes, this choice sacrifices some reproducibility, and doing something like generating a true random seed and writing it into a log file would be better, but all of this is additional mental overhead that for a good fraction of people will simply be too much. Anybody who has taught students knows that if you provide example code with a seed, some fraction of people will use your code as written and not change the seed. It doesn’t matter how often you say “change the seed.” If your code contains a seed, people will end up using that exact seed, every time.

There’s one more issue. In particular when you’re coding with scikit-learn, you need to set random seeds all over the place. Every single function that has random behavior has its own separate random number generator. So you can quickly face the situation where you need many random seeds. Want to do a train/test split? Please provide a random seed. Want to do a t-SNE? Please provide a random seed. Want to do a PCA? Please provide a random seed. Want to fit a random forest model? Please provide a random seed. You will quickly run into decision fatigue where you won’t have the energy to come up with new random seeds everywhere. You could set up an elaborate scheme where you have a master random number generator which you use to generate random seeds for each step of your analysis, but come on, nobody is going to do this. So I still think it is better to get into the habit of not setting a random seed at all and instead relying on the system random noise the library uses by default.

Let me end with this post on BlueSky by Ehud Karavani. It captures the right attitude. This is what you should be doing.

I cannot emphasize how bad of a choice this is. 42 was a joke guys. Don’t use 42. Ever. If your random seed is 42 I will come to your office and set your computer on fire.1

But is this actually a problem? Do people really use 42 that commonly as their random seed? Yes, absolutely. Google “random seed” or “random_state” and the number 42 will pop up among your top search hits. And people may explain where the number 42 comes from (we’ll get to this below), but then they don’t talk much about whether or not this choice is a good idea. In fact, frequently you see statements along the lines of “the random seed is arbitrary, you can use any number you want, so 42 is a fine choice.” This sentence is 100% correct, assuming you’re the only one who uses 42 and you also use it only once in your entire life. Obviously this assumption is not valid. But I rarely see people point this out.

For example, the documentation for the Python machine-learning framework scikit-learn contains a lot of material about random states and various options of controlling them. Everything the documentation says is technically correct, and yet it never discourages you from using 42 as the seed. In fact, the glossary contains this gem:

Popular integer random seeds are 0 and 42.

(I hope I won’t have to explain why 0 is just as bad a choice as 42.)

The number 42 also shows up throughout the documentation, such as in code examples for the train_test_split() function. Using a fixed random seed when splitting data into training and test sets is uniquely bad, as you’re always going to be sampling the same split when you’re re-training your classifier.

But it gets worse. LLMs have learned about 42 and will happily put it into the code they generate. Here is some zero-shot data-analysis code I recently generated. And there it is, random_state=42 (“for reproducibility”).

It is not surprising that attentive students, who read the documentation, read the blog posts, read the code generated by LLMs, conclude that a random seed of 42 is a good choice, and maybe even a choice that is superior to other options.

To dig deeper into why 42 is not a good choice, and is in fact uniquely bad, we need to look into how random numbers are generated in a computer.

What is a random seed?

In modern computing, we need randomness everywhere. If you’re doing machine learning and you need to subdivide your data into training and test sets, that requires a source of randomness. If you’re writing a computer game and you want your NPCs2 to show somewhat interesting and unpredictable behavior, you need randomness. If you’re simulating a physical system, you need randomness. The problem is that it’s rather complicated to generate true random numbers. The sources of true randomness we have available (for example from thermal fluctuations in specific electronics components) are nowhere fast or cheap enough to generate random numbers at the scale needed in modern computing environments.

The solution computer scientists have come up with is the pseudo-random number generator (PRNG). A PRNG is a mathematical algorithm that produces sequences of numbers statistically indistinguishable from random. Importantly, a PRNG will always produce the exact same sequence of numbers when run from the same starting point. The numbers aren’t random at all! But they look random.

So what is the random seed? It is a number that defines the initial state of the PRNG. How exactly we get from the seed to the initial state can be complicated, but the details don’t matter here. What matters is the same seed will always give you the exact same sequence of random numbers.

A second important concept to be aware of is the period length of a PRNG. The period length is the number of random values a PRNG can generate before it starts repeating. All PRNGs repeat eventually. Therefore, it is critical that your PRNG has a period length large enough that it never causes you any trouble. Let’s say you write a large numerical simulation (maybe you’re simulating the weather, or the early universe, or all the atoms in a cell) where you need trillions or more of random numbers. You wouldn’t want the random numbers to repeat during any of your simulation runs. So you need a PRNG with a period length well in excess of the maximum number of random values you may ever need.

One of the most widely used PRNGs is the Mersenne twister. It has a period length of over 10⁶⁰⁰⁰. (The exact value is 2¹⁹⁹³⁷ − 1.) This is an unimaginably large number. To give you a sense of how large it is, for comparison, there are approximately 10⁸⁰ atoms in the universe. This is tiny compared to 10⁶⁰⁰⁰. The period of the Mersenne twister has space for entire universes for every single atom in the universe, and then some. In fact, you could create an entire universe for every atom, and then create another entire universe for every atom in every of the universes you have created, and keep nesting 75 times, and still you wouldn’t run out of room in the period of the Mersenne twister. If you used the Mersenne twister to create nested universes 75 times deep, all these universes inside universes inside other universes would be different from each other.

What is a good choice for your random seed?

As I wrote above: The random seed is arbitrary. You can pick any seed you want. There are no better or worse seeds. (Unless you have a bad PRNG, but let’s ignore this complication.) In principle we could stop here. But in practice it’s a little more complicated.

While the seed is arbitrary, you don’t ever want to reuse a seed. The point of a PRNG is that its output is statistically indistinguishable from random. That’s going to be the case if you’re using a different seed every time. But if you’re reusing seeds, suddenly you have hidden correlation structures. And you may not even be aware of them.

The consequences of reusing random seeds could be benign or disastrous. It depends on the specifics of the situation. Let’s say you’re doing machine learning, and you’re using the train-test splitting code I quoted above, with a fixed random seed. In this case, you’re always splitting the data in exactly the same way. If you’re running this code five times, you’re not actually getting five independent splits, you’re getting the same split five times. At a minimum, that’s going to be wildly underestimating the variance in the performance of your fitted model. And worse consequences are possible if you’re unlucky.

If you’re following so far, and you’re starting to see that reusing random seeds can be bad, you may also realize that double-digit random seeds are bad. There are only 90 different options. If you’re in any way regularly working with random processes, you’ll easily need 90 different options in just a few days.

So, let’s go wild. Let’s use an 8-digit random seed.3 Surely that’ll give us sufficiently many different possibilities for a lifetime. Well, once you ponder it a bit, you’ll see that this doesn’t even give us a separate random sequence for every person in the United States. (The US population is approximately 340 million.) If they were all doing data science, splitting data into training and test, many of them would be using the exact same “random” splits.

The space you can possibly explore with even quite a long random seed is tiny compared to the total number of sequences you would want to have available to you, and which a PRNG such as the Mersenne Twister would certainly support.4 And the random seed 42 is uniquely bad precisely because everybody is using it. The Mersenne Twister has a state space large enough for universes within universes, but every data scientist in the entire world is using the same 10,000 “random” numbers that you get when starting with seed 42.

So we’re all on the same page, here are the first 50 of the “official” random numbers. They have been used millions of times. I encourage you to verify you get the same numbers on your computer.

>>> import random
>>> random.seed(42)
>>> [random.random() for i in range(50)]
[0.6394267984578837, 0.025010755222666936, 0.27502931836911926, 0.22321073814882275, 0.7364712141640124, 0.6766994874229113, 0.8921795677048454, 0.08693883262941615, 0.4219218196852704, 0.029797219438070344, 0.21863797480360336, 0.5053552881033624, 0.026535969683863625, 0.1988376506866485, 0.6498844377795232, 0.5449414806032167, 0.2204406220406967, 0.5892656838759087, 0.8094304566778266, 0.006498759678061017, 0.8058192518328079, 0.6981393949882269, 0.3402505165179919, 0.15547949981178155, 0.9572130722067812, 0.33659454511262676, 0.09274584338014791, 0.09671637683346401, 0.8474943663474598, 0.6037260313668911, 0.8071282732743802, 0.7297317866938179, 0.5362280914547007, 0.9731157639793706, 0.3785343772083535, 0.552040631273227, 0.8294046642529949, 0.6185197523642461, 0.8617069003107772, 0.577352145256762, 0.7045718362149235, 0.045824383655662215, 0.22789827565154686, 0.28938796360210717, 0.0797919769236275, 0.23279088636103018, 0.10100142940972912, 0.2779736031100921, 0.6356844442644002, 0.36483217897008424]

Should you explicitly set your random seed?

Why choose a specific random seed at all? Is this actually a good idea? In general, I think the answer is no. In my opinion, you’re typically better off using a random5 random seed and sampling a broader space of possibilities than picking your own random seed and risking that you’re drawing invalid conclusions from your analysis. However, there are of course specific situations in which picking a random seed is appropriate, or even required. Let’s discuss those.

Most importantly, in simulation studies, it can be helpful to start all simulations with a different but defined random seed, so that every single simulation run can be reproduced if necessary. In these types of situations, a good strategy is to take some arbitrary large integer number, say 9427385,6 and then add to it the number of the replicate you’re running. So, simulation i would have seed 9427385 + i. This requires a bit of work to get right, as you should never reuse any random seeds among simulations, but it can be useful for tracking down weird behaviors that may occur only occasionally.

Related to this point, when you’re coding a complex stochastic simulation, you may encounter bugs that happen only very rarely. Your simulation may work just fine most of the time, but every few thousand runs or so it crashes. This type of bug can be difficult to investigate. The first step is usually to identify a random seeds that reliably triggers the bug. If you can find a seed that triggers it early in the simulation run that’s even better.

Another scenario in which you might want to use defined random seeds is when you have random choices that you explicitly want to reuse multiple times. For example, if you’re doing machine learning, and you’re comparing two different models, you may want to fit them to the exact same collection of training/test splits. In this situation, you could, for example, pick ten random seeds, use each to generate one training/test split, and fit each model to each of the ten splits.

Finally, when making visualizations that contain random scatter or other elements that are randomly chosen, it can be helpful to play around with the random seed until the scatter looks pleasing. When I wrote my book on data visualization I used this technique quite frequently, for example in this chapter.

Where does 42 come from anyways?

Now that you know everything there is to know about random seeds, let’s go back to the number 42. Where does it come from? Why do people use this number in particular? The culprit is Douglas Adams’ The Hitchhiker’s Guide to the Galaxy, a humorous, quirky science fiction novel. The book was very popular in the 1980s and 1990s, was adapted several times for radio and TV, and reached broad audiences around the world. If you’ve never read it or seen any of the adaptations, I would encourage you to check it out. Just be prepared for rather strange humor.7

In the book, we learn about some hyper-intelligent, pan-dimensional beings that built a massive computer called Deep Thought, with the specific purpose to discover the answer to “the ultimate question of life, the universe, and everything.” After several million years of computing, Deep Thought reveals that the answer is 42. When the beings don’t understand what to do with this answer, Deep Thought tells them that to understand the answer they have to figure out what exactly the question is.8 You can see a film adaptation of this scene here.

The number 42 has since turned into a meme. When somebody asks an extremely broad question, or a question that goes after deep philosophical topics such as the meaning of life, people like to respond with 42. This meme is so popular it has its own Wikipedia article. This is all very funny,9 but none of this makes 42 a good random seed.

Updates

This post generated a number of good responses. I will collect the most helpful or interesting ones here.

First, Nick Bailey pointed out a good solution to the randomness/reproducibility conundrum (where reusing seeds ruins randomness but using true random starting points ruins reproducibility): Generate a genuinely random random seed when you start up your code, write it into a log file, and then seed your random number generator. This gives you the best of both worlds.

A somewhat similar idea: Use the current date as random seed. This at a minimum gives you a different seed each day, while also avoiding the problem of appearing to have fished for the seed that gives you desired results. The devil is in the details though of how exactly you convert the date into a seed. See this post by Stephen Turner and the subsequent replies:

Thanks to Colin Gillespie over on BlueSky, I have learned how to do code searches on GitHub. So now I can report that, as of this writing, there are 496k cases of random_state=42 on GitHub.

Finally, an issue to be aware of if you’re using Matlab: It uses a fixed random seed every time, so the generated random numbers are always the same in a fresh Matlab session:

You can read more about this on the Matlab blog.

To provide an alternative perspective, here I’m re-publishing a lightly edited version of a blog post from 2013, about how most graduate students propose to do too much and need to be reined in in their ambition. The post is primarily about the PhD thesis proposal, where students need to present a plan for their PhD work to a panel of professors. However, much of its content applies more broadly. Even after having successfully defended a thesis proposal many graduate students want to do too much.

I cannot remember ever having seen a graduate student present a PhD thesis proposal and be criticized for lack of ambition. It never happens. Even the weakest students—especially the weakest students—present proposals that are overly ambitious and that won’t ever get done, and certainly not in the 3–4 years remaining until graduation. In fact, in my experience it is exceedingly rare that a student presents a reasonable proposal, one that is actually doable during the remainder of their time in graduate school. Usually, those only happen when students “forget” to have their qualifying exams and end up presenting their “proposal” six months before the intended graduation date. In those cases, the students know that they won’t accomplish much new between proposal day and defense day, and therefore they present a proposal that consists entirely of completed work.

In biology PhD programs in the US, most professors expect graduate students to complete about three projects, corresponding to the magical three specific aims in a typical grant proposal.1 It follows that a graduate student who is defending their proposal, 2–3 years into their program, should have one project completed, one well under way, and one in the early planning stages. Students doing complicated experimental work might have progressed less, but at a minimum they should have one project well under way when they defend their thesis proposal. This leads to a pretty good rule of thumb for the amount of work the proposal should encompass: Aim 1 should be the work that is in the bag, and Aims 2 and 3 together should not require more than twice the amount of work already accomplished.

I rarely see PhD proposals that meet this rule of thumb. Instead, the already completed work is frequently only a small component of the proposed Aim 1, which by itself is going to take another two years to complete. Proposed Aim 2 will need four years on top of that, and Aim 3 another ten. Many graduate students propose to carry out a lifetime of research during their graduate studies.

I don’t quite know why PhD proposals tend to be overly ambitious. Maybe it’s youthful optimism or naiveté. I suspect, though, that there is a component of worry, the eternal graduate student concern of not being sufficiently productive, of not doing enough. Ironically, this concern often causes students to overlook the successes that are within reach and instead try to reach for the stars. In general, doing a successful thesis is a fine balancing act between being overly ambitious and playing it too safe, a topic for another post. However, there’s a difference between an actual PhD thesis and a thesis proposal: The ideal thesis will contain some exciting, risky work, but for the proposal most professors want to see a plan that is doable, not one that might be doable if the stars align correctly. As a smart graduate student, you have two alternative plans, one safe and one daring, you work on both of them at the same time, and you only present the safe one during the committee meeting.2

A second, related issue I frequently notice is that students display poor judgment in how much work they can realistically accomplish in their remaining time. Estimates are consistently too optimistic. If you are in year three, and you have completed 50% of your first project, it is unlikely that you’ll complete this and two entirely different projects in the remaining 2–3 years of your PhD. Further, unless you’re a paper-writing machine, it’s unlikely that you can write a paper in less than three months.3 So, if you still have three manuscripts to complete, plus a thesis, the writing time alone is going to be about a year. If you’re already halfway into year three, you’ll have about another 18 months of actual research work you can do, because the other 12 you’ll spend writing. (Of course I’d recommend that you don’t wait all the way till the end before you start writing, but the math comes out the same.) My personal rule of thumb is things take about three times longer than what students estimate. So if a student says a particular project needs another three months in the lab plus a month to be written up, I expect that project to be done around the same time next year.

In conclusion, when you prepare your thesis proposal, realistically assess how much work you can complete during the remainder of your graduate years. Don’t assume that your productivity will double or triple over the next two years, because it won’t. Budget at least three months for every paper you have to write, and triple the time you think it takes to complete the remaining lab work. If you have papers in review, consider that responding to reviewer comments and revising a paper frequently takes another two to three months, during which nothing else gets done. If you end up with a plan that will require another five years of work or more, then you’ll have to change your aims. See whether your current Aim 1 can be broken down into reasonable sub-aims which can be considered the separate chapters of your thesis. It’s quite common for me to conclude a PhD proposal defense by telling the student it’d be best to scrap Aims 2 and 3 altogether and instead expand Aim 1 into the entire thesis. If you come to this realization before the committee meeting, I won’t have to tell you so during, and everybody is happier.