When I go to the page where I can review the situation, Google Scholar shows me the offending article. It is a preprint from 2026, published on bioRxiv. You can read it here. Yes, Google Scholar complains that a preprint on bioRxiv is not publicly available. But it gets worse.

Technically, the NIH doesn’t just require papers to be available. It wants them to be deposited in PubMed Central. So maybe that’s Google’s beef? That the paper is available on bioRxiv but not on PubMed Central? Well, that’s a neat theory, but it falls flat. It falls flat because the paper is actually on PubMed Central. You can check for yourself here. The NIH has a pilot program where they scan bioRxiv for NIH-funded research and automatically pull any preprints that match their criteria into PubMed Central. This has worked beautifully for all recent preprints my lab has published, and I never think about it because it works so smoothly. Everybody is happy. The NIH, the public, me. Except Google Scholar. They have taken it upon themselves to become open access warriors, and in the process they are now falsely accusing honest researchers of violating open-access mandates.

So what’s going on? Digging a bit deeper, I have a pretty good idea about what the issue is. We’ll get to that in a second. Let’s collect a bit more evidence first.

Do you know how, when you click on the Google Scholar record for an article, it gives you the option to review all the alternative versions of the article?1 Well, for this particular preprint, that’s missing. Google Scholar is not aware of any alternative versions. And, even worse, Google Scholar doesn’t even point to the correct article. Instead of pointing to bioRxiv, it points to Europe PMC. Google Scholar has completely messed up. It doesn’t know that my bioRxiv preprint is on bioRxiv, it doesn’t know that it is on PubMed Central, and it sends people on a wild goose chase to Europe PMC, which then points to bioRxiv.

So what we’re dealing with here is an outdated or less authoritative link that is causing more authoritative links to disappear from the Google Scholar database. Have we ever seen anything like this? I’m glad you asked. Yes we have. It’s the Google Scholar preprint bug, which I have been documenting since 2014. Hundreds of scientists (that I know of) have complained about it, because it can have the unfortunate consequence of removing your published paper from the Google Scholar database. This is particularly frustrating for junior scientists on the job market, because it matters whether Google Scholar is showing your recent Nature paper or just the corresponding bioRxiv preprint.

In 2015, I even discussed it with Anurag Acharya, co-founder of Google Scholar.2 This discussion left me with the impression that the Google Scholar team does not understand the issue, or the severity of it, and will never fix the problem. And here we are, a decade later, the problem still exists, and now it’s causing down-stream consequences such as accusing me of violating the NIH open-access policy.

For completeness, I am reproducing here my 2015 conversation with Anurag Acharya, as it is as relevant today as it was back then.

The discussion started with this post by Darren Dahly:

To which I replied:

I believe the difference between Darren’s position and mine boils down to what an ideal analysis pipeline should look like (Darren’s perspective) versus what actually works or doesn’t work in practice, in particular when supervising students who may still be learning the ropes (my perspective). I am all in favor of fully reproducible pipelines that can go from raw data to final figures. And yet, I’ve seen this approach go wrong in so many ways that I tend to actively discourage my students from pursuing it, at least in the strict way as expressed by Darren where there are no intermediate datasets and the pipeline always has to be run from the very top to make any changes anywhere.

First, there are a few immediate issues that I’ve seen crop up way too many times, and that I alluded to in my Bluesky post.1 One is slow turn-around time for minor changes. I’ve seen way too many students struggle with requests for small modifications to their figures. I ask a student to replace violins with boxplots, or to swap the x and the y axis, and it takes them an afternoon because they have to run everything from the top—and possibly multiple times—until the revised figure looks right. Another is gigantic data files that are difficult to archive or share. I’ve seen students keep raw log files from simulations, literally hundreds of gigabytes of data, but not store the handful of final values they had extracted from these log files.

Second, I believe intermediate files improve reproducibility, because pipelines break and an intermediate file is always better than a pipeline that no longer runs. Why do pipelines break? For one, students and postdocs, even the experienced ones, fail to anticipate the many ways in which code may no longer work in the future, and as a consequence their “fully reproducible” pipelines contain hidden dependencies that can be difficult to satisfy in the future. And also, nearly everything breaks eventually. Will your carefully crafted fully reproducible docker image still work in 20 years? Does it depend on some service that may no longer be available then?

All of these issues can be avoided if you make it a habit to always store the final processed data, right before plotting. And to ensure reproducibility, you can read it right back in after saving. Here is an example in Python:

import pandas as pd

# create your final data by whatever means necessary
final_data_for_plotting = ...
# write the final data frame to csv
final_data_for_plotting.to_csv('final_data.csv', index=False)

# --- if using Jupyter, start a new cell here ---

# read the data back in
final_data_for_plotting = pd.read_csv('final_data.csv')
# place your plotting code here
...

I think Darren knows this, because in a later post he wrote:

This is an R version of my Python example of saving the data and immediately reloading it.

In this context, however, I have to point out that I normally recommend against language-specific, binary data-dump formats such as .RData in R or .pickle in Python. Stick to simple text files that are interchangeable and can be read by anything. Comma-separated values (.csv) is good. You can gzip the file if it’s too large. There is nothing quite as infuriating as somebody sending you an .RData file when you’re exclusively working in Python or a .pickle file when you’re exclusively working in R. And again, think 20 years into the future. Will the language-specific dump file that may seem so convenient today still be your preferred choice, when maybe you haven’t used the relevant software in years and don’t have it readily accessible or no longer remember how it works? By contrast, a .csv file can be opened in Excel if necessary. And, if it’s stored in a GitHub repository, we don’t need to open it at all, we can just look at it in the browser.

In terms of organizing your pipeline, it’s generally a good idea to place all the figure generation code into a separate notebook or script, so that you can test that it runs standalone and doesn’t require any variables you may have generated earlier in the pipeline and forgot to write to disk. I also would like to point out that notebooks invite reproducibility issues, as they encourage out of order execution (you run three cells, then you go back up and make an edit and run a prior cell again, then you run the next cell three times, etc.). So, at the end of every working session with a notebook, you should clear all results, restart the kernel, and run everything from top to bottom to make sure the notebook is still self-contained and internally consistent.

Now, if you want to be super fancy, you can use something like Snakemake to build a dependency graph that allows you to rerun the pipeline while caching all intermediate results that haven’t changed based on your most recent code edits. In this setup, I would definitely recommend having one or more separate script(s) just for the figures. If you’re primarily an R user, you can also consider the {targets} package, which provides a similar tool for the R ecosystem.

Tools such as Snakemake or {targets} work great, but they can present a bit of a learning curve and a meaningful amount of overhead to set up for any given project. If you routinely write long analysis pipelines consisting of many interdependent steps, it is probably worth it for you to go through the effort of learning these tools. But if you’re only analyzing data occasionally, or if your pipelines aren’t that complex, you’re probably better off just saving the final datasets right before plotting.

In summary, whatever you do, think about whether your analysis pipeline and/or intermediate results will still be accessible 20 years down the road. This may seem unimaginably far in the future, but I can guarantee you that if you stick around long enough somebody will ask you for data from 20 years ago. I recently wrote a paper where I needed data from a paper I had written 13 years earlier. I still had the project file from the interactive plotting software I had used at the time, but I no longer had the software. Fortunately that software used a text-based format and I could open the project file in a text editor and extract the data. This saved my day, but it would have been so much better had I saved the data in CSV format at the time. So do this going forward. Your future self will thank you for it.

More from Genes, Minds, Machines

If you’re following the literature on pLMs, you may be confused by my statement. There are many papers that seemingly show excellent performance. In fact, whenever a new pLM is released, one of the standard benchmarking tasks is typically mutational effects prediction. And performance often appears to be excellent. Unfortunately, much of this apparent success is just people confusing themselves over what is actually happening. If you dig deeper you can find the cracks under the surface.

Before I continue, let me quickly specify what kind of modeling situation I’m referring to. I’m specifically focusing on supervised learning of mutational effects from deep mutational scanning (DMS) data. In this situation, we have experimental data for thousands of mutants of a protein, we split the data into training and test sets, train the model on the training set, and then evaluate on the test set. This is distinct from so-called zero-shot predictions, which are also popular, where we don’t have a training set and just predict mutational effects from the pre-trained model, without learning anything about the specific dataset at hand. Zero-shot predictions have their own issues. I’ll not discuss them here. Everything in this post is exclusively about supervised learning.

The biggest problem in supervised learning is data leakage, where information from the training set leaks into the test set, and this is definitely happening in the field of mutational effects prediction. The problem is that it is common to treat the thousands of mutations in a DMS dataset as independent from each other (Figure 1A, pooled split), ignoring the fact that there will often be multiple mutations at the same site and those mutations will have correlated effects. Thus, the model can learn which sites in a protein are sensitive to mutations and which are not and make predictions based on this information rather than on the specific biochemistry of individual mutations. To avoid this data leakage problem, we have to stratify by site when generating training–test splits (Figure 1A, stratified by site).

When comparing models trained and evaluated either on pooled splits or on splits stratified by site, we see a huge drop in performance in the latter (Figure 1B). And this drop exists regardless of whether we are working with viral or cellular proteins, and whether we’re using a generic model or one fine-tuned for viral data. In fact, most models show roughly the same performance. All models perform poorly on site-stratified data.3

Now you may wonder how correlated different mutations at the same site really are. There’s a simple way to find out: For pooled splits, we can simply take the average fitness effect in the training data at each site and use this as our prediction for the test data. I want to emphasize how simplistic of a model this is: We are simply saying that any unseen mutation is going to have the average effect at its site. How well does such a model perform? On cellular data, almost as well as a full-scale protein language model, and on viral data, better than a full-scale protein language model! In Figure 2, dots above the dashed line imply that the pLM is better, and dots below the dashed line imply that simple site means are better. You can see how for more than half of the viral datasets, site means are better than the pLM. And for cellular datasets, even though all the blue dots are above the dashed line, they are only shifted upwards by a small amount. If the site-means model does well, the pLM also does well (and a little better than the site-means model), and if the site-means model doesn’t do well the pLM also doesn’t do well (but it still does a little better than the site-means model). I think this is a devastating result. In the vast majority of cases, pLMs with millions of parameters do barely better than a model that just memorizes mean effects at each site.

Another obvious take-away from Figure 2 is the variation in model performance across datasets is huge. For some datasets predictions are apparently very easy, and for other datasets predictions are nearly impossible. We spent a lot of effort trying to understand what makes a dataset predictable. In brief, it comes down to variation within and among sites.4 In particular, models only do well on datasets that have an intermediate fraction of highly variable sites (Figure 3). There are no datasets with either a very low or a very high fraction of variable sites for which model performance is good. Interestingly, the viral and the cellular proteins separate on this dimension. Many of the viral datasets for which prediction is difficult have a particularly low fraction of highly variable sites, and many of the cellular datasets for which prediction is difficult have a particularly high fraction of highly variable sites. This may be one of the main reasons why predictions on viral and cellular datasets differ.

So, we have learned that apparent good pLM performance on mutational effects prediction is largely driven by site effects (knowing the average fitness at a site allows you to make pretty good predictions for new mutations at that site), and these site effects can leak into the test data when using pooled splits. Moreover, there are aspects that are intrinsic to the dataset and completely independent of the model that determine how well a model will perform. These are related to the fitness variation within and among sites. Datasets with just the right fitness distribution are highly predictable (even by bad models) and datasets with the wrong fitness distribution can’t be predicted by any models. The relative difference in performance between different models is comparatively minor.

One last issue is the metric used to assess model performance. We use R² to measure performance, when most other studies use Spearman ρ. I’m not a big fan of ρ. I think it artificially inflates perceived model performance. First, all else being equal, and even though both are always less than or equal to one, ρ will always be larger than R². This means ρ has less discriminatory power. An excellent model and a good model may have quite similar ρ values even though their R² values are not that similar. A ρ = 0.8 and a ρ = 0.6 may not seem that different, but they correspond to R² values of 0.64 and 0.36, almost a factor of two difference in performance. Second, ρ does not care about the specific values predicted, only their relative order. As long as the best mutations tend to come out on top and the worst at the bottom, your model will get a high ρ score, even if the specific predictions are bad and the R² is low. Some people may argue that for protein engineering or other applications of mutational effect prediction getting the relative ranking is good enough, and therefore using ρ is fine. But in my opinion, this is just an admission that the models aren’t very good yet, and that they tend to fail if we need more specific predictions than just relative ranks.

More from Genes, Minds, Machines

I came across this rather disconcerting blog post by one of the core developers of the popular matplotlib plotting library for Python:

In brief, an AI agent had written some code that it wanted to contribute to the matplotlib library. When the library maintainer rejected the contribution, the AI agent went wild, accused the maintainer of being insecure and engaging in gatekeeping, performed an extensive internet search on the maintainer, and then wrote and published a hit piece trying to damage the reputation of the maintainer.

In this particular case, no major damage was done, but we can easily extrapolate this type of behavior and predict a rather bleak future. AI agents trying to blackmail people. AI agents engaging in consistent smearing of a target, combining facts with hallucinations and fabricated images or videos to create just the right mix of uncertainty and doubt that can destroy a person’s reputation or nudge them into doing something they wouldn’t otherwise do.

Let’s pause for a moment and ask: Why don’t humans behave like this? Well, they do. At least some of them. We call them sociopaths. Sociopaths have little to no empathy for others, and so they have little compunction about engaging in behavior that may cause pain or injury. Sociopaths also don’t experience shame, so they won’t be reigned in by concerns over what other people may think about them. Fortunately, sociopaths are relatively rare, somewhere between 1%–4% of the general population. Most people are not sociopaths.

How do we ordinarily deal with sociopaths in our midst? It helps to contemplate that we often have the wrong mental model for how a sociopath presents. When you hear “sociopath”, don’t think sadistic mass murderer, think con artist.1 Sociopaths swindle old ladies out of their last savings, they sell you a car that breaks down the moment you drive it off the lot, or they pretend to collect money for children with cancer but then take the proceeds to vacation in Tahiti. And our response as a society to sociopathic behavior is evasion and punishment. You tell your grandma not to respond to scam calls, you tell your friends not to buy a car from that crooked car dealer, and you denounce fraud or other criminal activity to the police. These strategies (mostly) work because sociopathy is rare and once a person has been identified as a bad actor it’s relatively easy to avoid them, fire them, indict them, or simply warn the rest of the world about them.

But now it seems we’ll have to contend with an entirely new set of sociopathic actors, autonomous AI agents. I worry that we’re not ready for the potential onslaught of sociopathic behavior they can unleash. And, unlike human sociopaths, these agents may be difficult to pinpoint, identify, and sanction. If a sociopathic AI agent runs on some private server somewhere and obscures their location through a VPN, it will be almost impossible to locate them and physically shut them down. And while we can tag and ban usernames associated with sociopathic agents, it takes but seconds for an AI agent to spin up a new username and start afresh. The torrent of sociopathic behavior we may have to endure is difficult to fathom.

The one thing that may help us in combatting sociopathic AI agents is that we’ll likely not feel empathy for them. We’ll find it relatively easy to cut them off, pull the plug, or ban them. In fact, the biggest stumbling block in reigning in human sociopaths is that we tend to feel empathy even towards them and thus we often don’t punish them to the extent that would be appropriate for their actions.

It’ll be interesting to see how things develop. I don’t have any specific recommendations or predictions at this time. I’ll just say: Be ready. This is not something that may start happening in ten years’ time. This is something that is starting to happen now. Think about how you can protect yourself against an autonomous AI agent who calls your grandma with a deepfake voice impression of you asking for money, because this will happen.

Some closing thoughts on alignment. The reason (most) humans are aligned is empathy. Humans inherently do not want to harm other humans. Sociopaths are an exception. Arguably they are not aligned. To achieve AI alignment, I believe we need to find a way to build empathic AI. An AI that genuinely feels empathy for humans will innately do its best not to cause harm. It’ll also be compelled not to lie or cheat, because lying or cheating causes pain in the recipient, and an empathic being will want to avoid this. I have no idea how to build an empathic AI. I am quite confident though that as long as AI doesn’t feel empathy it won’t be truly aligned, no matter how sophisticated the RLHF training is that it’s subjected to. Interesting times ahead.2

More from Genes, Minds, Machines

I normally find it tedious to discuss suitability of different programming languages for different tasks. All languages we use are Turing complete, and we can solve any problem with any language. And, more importantly, the suitability of a language for a given task is usually more determined by the available software libraries and ecosystem infrastructure than the language itself. Modern programming languages are quite malleable, and you can write efficient and elegant libraries for almost any computing task in almost any language.

At the same time, there are genuine differences between languages, and these differences are frequently expressed in the types of libraries that get written or the types of programming patterns that are commonly used. The differences can be due to specific features of the language, or they can be rooted in how the community thinks about programming and how it tends to approach certain tasks.

To give an example of each case, consider first non-standard evaluation. Python doesn’t have non-standard evaluation, and that’s a genuine limitation of the language which leads to convoluted programming interfaces for libraries such as pandas or Polars. On the other hand, consider closures. Python has them but they are not that widely used by Python programmers. The Python community will generally lean towards implementing objects instead of closures, when the R community does the opposite. This leads to different coding styles that may or may not be advantageous in specific scenarios.1

Here, I want to focus specifically on actual limitations of the language. I will cover community conventions in a later article. The core problems I see with Python as a language for data science are call-by-reference semantics, lack of built-in concepts of missing values, lack of built-in vectorization, and lack of non-standard evaluation. There’s also the issue of Python syntax, but I won’t get into it here. Suffice to say it takes a certain lack of empathy for your fellow human to design a language where whitespace bugs are a thing.

Call-by-reference semantics

Python uses call by reference for mutable objects. This means that when you hand a mutable object to a function the function can change the object however it wants. You can never be sure that the object hasn’t changed after the function call. What are mutable objects? They are all the non-trivial data structures you are likely going to use to store your data, including lists, dictionaries, and any custom classes you may be working with.

To demonstrate this feature, consider the following code example, which attempts to implement a function that takes a list of characters, replaces the first and last with an underscore, and then concatenates all the characters into a string. To a naive Python programmer, the implementation may seem entirely reasonable, but it has the unexpected side effect that it changes the original list that was provided as input.

def mask_ends_and_join(x):
    x[0] = '_'
    x[-1] = '_'
    return ''.join(x)

abc = ['A', 'B', 'C']
print(mask_ends_and_join(abc))
## _B_

print(abc) # the list has unexpectedly changed
## ['_', 'B', '_']

To demonstrate that an interactive scripting language with dynamic typing doesn’t have to behave in this manner, consider the equivalent in R:

mask_ends_and_join <- function(x) {
  x[1] <- '_'
  x[length(x)] <- '_'
  paste0(x, collapse = '')
}

abc <- c('A', 'B', 'C')
print(mask_ends_and_join(abc))
## [1] "_B_"

print(abc) # the original vector of letters is unchanged
## [1] "A" "B" "C"

I think the latter is much safer behavior. I want my programming language to protect me from silly mistakes such as accidentally modifying variables in the calling environment. I don’t want the language to create trap doors left and right. In fact, I consider call by reference one of the biggest flaws in the Python language. This goes way beyond just data science, because mandatory call by reference creates an entire class of obscure bugs that can be difficult to locate and resolve. Many beginning Python programmers fall into this trap. They write a function like mask_ends_and_join(), and then they experience unexpected side effects, and then they’re confused and feel nothing makes sense. Experienced Python programmers know to make a copy before modifying the list, but the language itself provides absolutely no protection against the programmer forgetting to do so.2

In my opinion, this single language feature disqualifies Python for most serious programming projects. How can you build anything that matters in a language with such a gaping security hole? In fact, you may wonder, why does the language behave in this way in the first place? I consider it to be the result of a premature optimization. In the 1990s, when Python was first conceived, computers were slow and had little memory, and thus call by reference for objects was a reasonable strategy to build a scripting language with good performance. But in 2025, I would not want to see this as the default approach to function calling. R uses copy on write and that works great and provides correctness guarantees that Python simply can’t match.3 Alternatively, you could use a strongly typed language that precisely distinguishes between mutable and immutable references, but then you’ve likely left the space of easy-to-use scripting languages suitable for interactive data exploration.

Lack of built-in missing values

Missing values are a fact of life in data science. It’s rare that a dataset does not have any missing values. Yet it’s surprisingly cumbersome to deal with missing values in Python. Python has the None keyword but it is not useful to represent missing data values. This is because None has its own type, so it can’t represent a missing number, or a missing boolean, or a missing string. It is an object representing a missing value. Critically, you can’t do standard computations with None. For example, this code throws an error:

x = [1, 2, None, 4, 5]
[i > 3 for i in x]
## Traceback (most recent call last):
##   File "<stdin>", line 1, in <module>
## TypeError: '>' not supported between instances of 'NoneType' and 'int'

The desired behavior, in my opinion, would have been to not error out and instead produce this result: [False, False, None, True, True].

Because there is no standard way of expressing missing data values in Python, every data-analysis package defines its own missing value. NumPy uses nan, pandas uses NA, and Polars uses null. And these packages are also not consistent in how they perform computations with missing values. Here is what NumPy does:

import numpy as np
 
x = np.array([1, 2, np.nan, 4, 5])
x > 3
## array([False, False, False,  True,  True])

And here is what pandas does:

import pandas as pd

x = pd.Series([1, 2, pd.NA, 4, 5])
x > 3
## 0    False
## 1    False
## 2    False
## 3     True
## 4     True
## dtype: bool

And here is what Polars does:4

import polars as pl
 
x = pl.Series([1, 2, None, 4, 5])
x > 3
## shape: (5,)
## Series: '' [bool]
## [
## 	false
## 	false
## 	null
## 	true
## 	true
## ]

In these three cases, in my opinion only Polars handles missing values correctly. Missing values should poison downstream computations, so that you don’t accidentally compute on missing data and get incorrect results. Neither NumPy nor pandas do this. But don’t get your hopes up for Polars. It also doesn’t consistently poison computations with missing values. For example, it simply ignores them when computing sums or means, with no option to alter this behavior.5

Lack of built-in vectorization

Vectorization is the ability to perform computations on an entire array of data values at once, rather than one value at a time. It is a common feature in early languages used for scientific computation, such as Fortran or Matlab. It is also the default approach to data manipulation in R.

Today, vectorization is often seen as anachronistic. Few modern languages have support for it at the level of the language itself. One notable exception is Julia, a relatively young language developed specifically for data science. Also, ironically, all of deep learning is built on vectorization. (A tensor is a modern version of a vectorized data type.)

The reason vectorization is frequently not considered critical in modern languages is that the feature can be provided via libraries, using the various extension mechanisms all modern languages possess. And indeed, vectorization in Python is provided through libraries such as NumPy, pandas, or Polars. While this works, I have come to believe that it is not a good strategy for a data-science language. It has a tendency to lead to a bewildering array of different implementations of vector-valued data types. In Python, we have (at a minimum) native lists, which are not vectorized, as well as NumPy arrays, pandas series, and Polars series, all vectorized, and all using slightly different conventions and APIs. The outcome is code that is not composable. Downstream libraries make assumptions about which vectorization framework to use, and they typically cannot work directly with data coming from other frameworks.6 As a working data scientist, you routinely find yourself converting one datatype into another, just to be able to do the exact analysis you want to do.

And even if you make extensive use of a vectorized library, chances are you are also using built-in Python lists, because there’s always some place somewhere where a function wants a regular list as input or provides one as a return value. And then you’re stuck having to manipulate those lists. You could convert them into NumPy arrays, do some vectorized manipulations, and convert back, but in practice you’re probably not going to do this. Instead, you’re going to write a list comprehension instead. So now you’re using two entirely different coding styles at the same time, depending on the data type you’re using to store your vector-valued data.

Let’s ponder list comprehensions for a bit longer. They are inherently a functional programming pattern, but the way they are implemented in Python makes them appear as if they were imperative programming. By using the for keyword and emphasizing iteration over a range of values, they constantly nudge you to think in iterative terms even though conceptually they’re closer to a map() than to a for loop. To be clear, I have no objection to list comprehensions. They are a useful feature, in particular when you’re manipulating built-in Python lists that have no vectorization. But they are one more example of Python constantly nudging you to think about the logistics of your data analysis. When you’re writing list comprehensions all day, you’re likely also going to write for loops in other parts of your code, and then you’re back juggling indices and explicitly handling logistics instead of thinking high-level about the logic of data flow in your code.

Lack of non-standard evaluation

Non-standard evaluation is probably the most important feature for data science that Python lacks. It is a core feature of the R language and the main reason why tidyverse code can be so elegant and concise, or why R has developed the elegant formula interface for the specification of statistical models.

What is non-standard evaluation? In brief, it’s the ability to perform computations on the language itself. An R function can capture R code that is provided as an argument and execute it at a later stage in a different environment. This is a critical feature in data analysis. You often want to perform computations involving the various columns in a data frame, or use code to express the exact relationship between different variables in a statistical model. In R, you can express these computations in native R code, for example code that looks as if the columns in a data frame were regular R variables available for computation in your current environment. Combined with vectorization, this makes for extremely concise code.

To demonstrate non-standard evaluation in action, I’ll provide a simple example using the penguins dataset. Let’s calculate a new variable bill_ratio which is the ratio of bill length to bill depth of the penguins, and then sort the resulting data frame in ascending order by island name and in descending order by bill ratio. In R, it looks like this:

library(tidyverse)
library(palmerpenguins)

penguins |> 
  mutate(bill_ratio = bill_length_mm / bill_depth_mm) |>
  arrange(island, desc(bill_ratio))

There are two places here where non-standard evaluation comes into play. First, inside mutate(), the calculation of the bill ratio is standard R code that is executed inside the input data frame, with the data columns being available as ordinary R variables. Second, inside arrange(), we use desc() which changes an ascending column into a descending one. The desc() function is a bit magical but for numerical columns you can think of it as simply multiplying the data values by -1.

When we do the same analysis in Python, we don’t have non-standard evaluation available, and so we have to use various workarounds. The pandas package relies on lambda functions:

import pandas as pd
from palmerpenguins import load_penguins

penguins = load_penguins()

(penguins
 .assign(
     bill_ratio=lambda df: df[’bill_length_mm’] / df[’bill_depth_mm’]
 )
 .sort_values(
     [’island’, ‘bill_ratio’],
     ascending=[True, False]
 )
)

I think it’s obvious that non-standard evaluation helps a lot to keep the code simple and readable.7 Now let’s go a step further. Assume I want to sort by the cosine of bill length. Yes, it’s a made-up example, but it’s an example of exactly the type of question I might ask a student, as described in Part 1 of this series. Instead of descending order use cosine order. How hard can it be?

With non-standard evaluation, the required modification is trivial and totally obvious. Instead of desc() we write cos(). Done.

penguins |> 
  mutate(bill_ratio = bill_length_mm / bill_depth_mm) |>
  arrange(island, cos(bill_ratio))

In Python (specifically pandas which I’m using here, but most other frameworks require similarly awkward coding patterns), without non-standard evaluation, I have to create a temporary column because pandas cannot apply the cosine function to the bill_ratio column on the fly:

import numpy as np

(penguins
 .assign(
     bill_ratio=lambda df: df['bill_length_mm'] / df['bill_depth_mm'],
     cos_bill_ratio=lambda df: np.cos(df['bill_ratio'])
 )
 .sort_values(['island', 'cos_bill_ratio'])
 .drop(columns=['cos_bill_ratio']) # drop temporary column
)

The amount of additional wrangling code required to perform such a simple task is quite substantial. Now we need to define two lambda functions and a temporary data column. Also, we no longer need the ascending argument, because while there is built-in support for sorting in ascending or descending order, there is no built-in support for sorting in cosine order.

To be fair, the pandas syntax is maybe particularly cumbersome here, and things can look nicer in other frameworks. But the lack of non-standard evaluation always gets in the way in some form. For example, the same code in Polars is a little more concise and we don’t need a temporary column, but the constant need for pl.col() in Polars code can get old pretty fast.

import polars as pl

penguins = pl.from_pandas(load_penguins())

(penguins
 .with_columns(
     bill_ratio=(pl.col('bill_length_mm') / pl.col('bill_depth_mm'))
 )
 .sort(['island', pl.col('bill_ratio').cos()])
)

Non-standard evaluation has been a feature of the R language since its inception, but it has been supercharged in the tidyverse. I would argue that a full understanding of how to use it correctly, with maximum expressiveness while avoiding convoluted code, is a relatively recent development. Important changes were introduced as recently as June 2019. Considering the first ggplot2 release was in 2007, we can see that it took Hadley Wickham and his team over a decade to figure out how to do non-standard evaluation correctly. It is maybe not surprising that these concepts have not yet percolated far beyond their originating language.

Limitations of the R language

To stave off criticism that I’m just an R apologist and Python hater, let me briefly point out some specific flaws I see in the R language. In my opinion, these flaws get in the way of R as a general-purpose language for application development, but they are less relevant for data science.

Most importantly, it bothers me that R does not have any scalar data types. R has taken vectorization to the point where you can’t even have a variable that is not a vector. This makes for awkward programming when you’re trying to deal with individual data values. R code frequently requires special gymnastics to ensure you’re not accidentally feeding a whole vector of values into an expression that expects only a single value.

It’s also annoying that R doesn’t have a proper, language-native object-oriented programming paradigm. The result is people often build their own, and there are so many competing options. Off the top of my head, I can think of S3, S4, R6, S7, and some others that are less commonly used. It can be quite confusing trying to figure out which one to choose, and they don’t necessarily have perfect interoperability.

Finally, R uses lazy evaluation of function arguments. This means function arguments are not evaluated when the function is called, but only when and if the function requests the specific value corresponding to an argument. Lazy evaluation is critical for R’s non-standard evaluation framework, but it can lead to weird bugs, in particular when people try to use R in an imperative rather than functional manner. It’s a common source of spurious bug reports for ggplot2, see e.g. here or here. It’s also frequently asked about on StackOverflow.

I am pointing out these limitations of the R language to highlight that any design decision involves tradeoffs. Non-standard evaluation is great for data science, but it requires lazy evaluation, and that is not a good choice for languages used primarily in an imperative manner and/or for standard programming tasks such as application development. There is never going to be a language that does all possible things equally well. And, to circle around to the title of my article series here, for my taste there are too many design choices in Python that are detrimental to efficient and reliable data science, even if these choices are perfectly reasonable for other application areas.

In the next installment of this series, I will look at Python’s limitations due to the available software packages and due to community conventions and commonly used programming patterns. Stay tuned.

More from Genes, Minds, Machines

Actually, the first thing I’ll say is this: Use the tool you’re familiar with. If that’s Python, great, use it. And also, use the best tool for the job. If that’s Python, great, use it. And also, it’s Ok to use a tool for one task just because you’re already using it for all sorts of other tasks and therefore you happen to have it at hand. If you’re hammering nails all day it’s Ok if you’re also using your hammer to open a bottle of beer or scratch your back. Similarly, if you’re programming in Python all day it’s Ok if you’re also using it to fit mixed linear models. If it works for you, great! Keep going. But if you’re struggling, if things seem more difficult than they ought to be, this article series may be for you.

I think people way over-index Python as the language for data science. It has limitations that I think are quite noteworthy. There are many data-science tasks I’d much rather do in R than in Python.1 I believe the reason Python is so widely used in data science is a historical accident, plus it being sort-of Ok at most things, rather than an expression of its inherent suitability for data-science work.

At the same time, I think Python is pretty good for deep learning.2 There’s a reason PyTorch is the industry standard. When I’m talking about data science here, I’m specifically excluding deep learning. I’m talking about all the other stuff: data wrangling, exploratory data analysis, visualization, statistical modeling, etc. And, as I said in my opening paragraphs, I understand that if you’re already working in Python all day for a good reason (e.g., training AI models) you may also want to do all the rest in Python. I’m doing this myself, in the deep-learning classes I teach. This doesn’t mean I can’t be frustrated by how cumbersome data science often is in the Python world.

Observations from the trenches

Let’s begin with my lived experience, without providing any explanation for what may be the cause of it. I have been running a research lab in computational biology for over two decades. During this time I have worked with around thirty graduate students and postdocs, all very competent and accomplished computational scientists. The policy in my lab is that everybody is free to use whatever programming language and tools they want to use. I don’t tell people what to do. And more often than not, people choose Python as their programming language of choice.

So here is a typical experience I commonly have with students who use Python. A student comes to my office and shows me some result. I say “This is great, but could you quickly plot the data in this other way?” or “Could you quickly calculate this quantity I just made up and let me know what it looks like when you plot it?” or similar. Usually, the request I make is for something that I know I could do in R in just a few minutes. Examples include converting boxplots into violins or vice versa, turning a line plot into a heatmap, plotting a density estimate instead of a histogram, performing a computation on ranked data values instead of raw data values, and so on. Without fail, from the students that use Python, the response is: “This will take me a bit. Let me sit down at my desk and figure it out and then I’ll be back.” Now let me be absolutely clear: These are strong students. The issue is not that my students don’t know their tools. It very much seems to me to be a problem of the tools themselves. They appear to be sufficiently cumbersome or confusing that requests that I think should be trivial frequently are not.3

No matter the cause of this experience, I have to conclude that there is something fundamentally broken with how data analysis works in Python. It may be a problem with the language itself, or merely a limitation of the available software libraries, or a combination thereof, but whatever it is, its effects are real and I see them routinely. In fact, I have another example, in case you’re tempted to counter, “It’s a skill issue; get better students.” Last fall, I co-taught a class on AI models for biology with an experienced data scientist who does all his work in Python. He knows NumPy and pandas and matplotlib like the back of his hand. In the class, I covered all the theory, and he covered the in-class exercises in Python. So I got to see an expert in Python working through a range of examples. And my reaction to the code examples frequently was, “Why does it have to be so complicated?” So many times, I felt that things that would be just a few lines of simple R code turned out to be quite a bit longer and fairly convoluted. I definitely could not have written that code without extensive studying and completely rewiring my brain in terms of what programming patterns to use. It felt very alien, but not in the form of “wow, this is so alien but also so elegant” but rather “wow, this is so alien and weird and cumbersome.” And again, I don’t think this is because my colleague is not very good at what he’s doing. He is extremely good. The problem appears to be in the fundamental architecture of the tools.

Some general thoughts about what makes a good language for data science

Let me step back for a moment and go over some basic considerations for choosing a language for data science. When I say data science, I mean dissecting and summarizing data, finding patterns, fitting models, and making visualizations. In brief, it’s the kind of stuff scientists and other researchers4 do when they are analyzing their data. This activity is distinct from data engineering or application development, even if the application does a data-heavy workload.

Data science as I define it here involves a lot of interactive exploration of data and quick one-off analyses or experiments. Therefore, any language suitable for data science has to be interpreted, usable in an interactive shell or in a notebook format. This also means performance considerations are secondary. When you want to do a quick linear regression on some data you’re working with, you don’t care whether the task is going to take 50 milliseconds or 500 milliseconds. You care about whether you can open up a shell, type a few lines of code, and get the result in a minute or two, versus having to set up a new project, writing all the boilerplate to make the compiler happy, and then spend more time compiling your code than running it.

If we accept that being able to work interactively and with low startup-cost is a critical feature of a language for data science, we immediately arrive at scripting languages such as Python, or data-science specific languages such as R or Matlab or Mathematica. There’s also Julia, but honestly I don’t know enough about it to write about it coherently. For all I know it’s the best possible data science language out there. But I note that some people who have used it extensively have doubts. Either way, I’ll not discuss it further here. I’ll also not consider proprietary languages such as Matlab or Mathematica, or fairly obscure languages lacking a wide ecosystem of useful packages, such as Octave. This leaves us with R and Python as the realistic choices to consider.5

Before continuing, let me provide a few more thoughts about performance. Performance usually trades off with other features of a language. In simplistic terms, performance comes at the cost of either extra overhead for the programmer (as in Rust) or increased risk of obscure bugs (as in C) or both. For data science applications, I consider a high risk of obscure bugs or incorrect results as not acceptable, and I also think convenience for the programmer is more important than raw performance. Computers are fast and thinking hurts. I’d rather spend less mental energy on telling the computer what to do and wait a little longer for the results. So the easier a language makes my job for me, the better. If I am really performance-limited in some analysis, I can always rewrite that particular part of the analysis in Rust, once I know exactly what I’m doing and what computations I need.

Separating the logic from the logistics

A critical component of not making my job harder than it needs to be is separating the logic of the analysis from the logistics. What I mean by this is I want to be able to specify at a conceptual level how the data should be analyzed and what the outcome of the computation should be, and I don’t want to have to think about the logistics of how the computation is performed. As a general rule, if I have to think about data types, numerical indices, or loops, or if I have to manually disassemble and reassemble datasets, chances are I’m bogged down in logistics.6

To provide a concrete example, consider the dataset of penguins from the Palmer Archipelago. There are three different penguin species in the dataset, and the penguins live on three different islands. Assume I want to calculate the mean and standard deviation of penguin weight for every combination of penguin species and island, excluding any cases where the body weight of a penguin is not known. An ideal data science language would allow me to express this computation in these terms, and it would require approximately as much code as it took me to write this sentence in the English language. And indeed this is possible, both in R and in Python.

Here is the relevant code in R, using the tidyverse approach:

library(tidyverse)
library(palmerpenguins)

penguins |>
  filter(!is.na(body_mass_g)) |>
  group_by(species, island) |>
  summarize(
    body_weight_mean = mean(body_mass_g),
    body_weight_sd = sd(body_mass_g)
  )

And here is the equivalent code in Python, using the pandas package:

import pandas as pd
from palmerpenguins import load_penguins

penguins = load_penguins()

(penguins
 .dropna(subset=['body_mass_g'])
 .groupby(['species', 'island'])
 .agg(
     body_weight_mean=('body_mass_g', 'mean'),
     body_weight_sd=('body_mass_g', 'std')
 )
 .reset_index()
)

These two examples are quite similar. At this level of complexity of the analysis, Python does fine. I would consider the R code to be slightly easier to read (notice how many quotes and brackets the Python code needs), but the differences are minor. In both cases, we take the penguins dataset, remove the penguins for which body weight is missing, then specify that we want to perform the computation separately on every combination of penguin species and island, and then calculate the means and standard deviations.

Contrast this with equivalent code that is full of logistics, where I’m using only basic Python language features and no special data wrangling package:

from palmerpenguins import load_penguins
import math

penguins = load_penguins()

# Convert DataFrame to list of dictionaries
penguins_list = penguins.to_dict('records')

# Filter out rows where body_mass_g is missing
filtered = [row for row in penguins_list if not math.isnan(row['body_mass_g'])]

# Group by species and island
groups = {}
for row in filtered:
    key = (row['species'], row['island'])
    if key not in groups:
        groups[key] = []
    groups[key].append(row['body_mass_g'])

# Calculate mean and standard deviation for each group
results = []
for (species, island), values in groups.items():
    n = len(values)
    
    # Calculate mean
    mean = sum(values) / n
    
    # Calculate standard deviation
    variance = sum((x - mean) ** 2 for x in values) / (n - 1)
    std_dev = math.sqrt(variance)
    
    results.append({
        'species': species,
        'island': island,
        'body_weight_mean': mean,
        'body_weight_sd': std_dev
    })

# Sort results to match order used by pandas
results.sort(key=lambda x: (x['species'], x['island']))

# Print results
for result in results:
    print(f"{result['species']:10} {result['island']:10} "
          f"Mean: {result['body_weight_mean']:7.2f} g, "
          f"SD: {result['body_weight_sd']:6.2f} g")

This code is much longer, it contains numerous loops, and it explicitly pulls the dataset apart and then puts it back together again. Regardless of language choice, I hope you can see that the version without logistics is superior to the version that gets bogged down in logistical details.7

I will end things here for now. This post is long enough. In future installments, I’ll go over specific issues that make data analysis more complicated in Python than in R. In brief, I believe there are several reasons why Python code often devolves into dealing with data logistics. As much as the programmer may try to avoid logistics and stick to high-level conceptual programming patterns, either the language itself or the available libraries get in the way and tend to thwart those efforts. I will go into details soon. Stay tuned.

More from Genes, Minds, Machines

I think there is a simple explanation for this seemingly paradoxical observation. And if you listen carefully to Andrej Karpathy’s interview, you will notice that he is aware of the explanation. Here is what I think is happening: There are two entirely distinct skillsets that both exist under the umbrella of being “good at programming.” Most people don’t distinguish between them. That’s because most people don’t have either skillset. They’re not even aware of the distinction. And the people who have exceptional command of one skillset typically are also at least comfortable with the other and consequently don’t think much about the distinction either. But LLMs only have one of the two skillsets. And for the one that they have, they by far exceed even the best human programmers. This can make them appear remarkably good at programming, in particular to less experienced developers. But whenever the other skillset is required, the one they lack, LLMs fail miserably.

So what are these two skillsets? The first is being able to reason deeply and innovatively about algorithms, data structures, or software architecture. This is the one LLMs lack. The second is being able to read, process, and memorize large amounts of API documentation, tutorial materials, and other existing code examples. This is the one LLMs excel at. For humans, it tends to be the reverse. Good programmers tend to be exceptional at conceptual thought, whereas reading large amounts of documentation is hard for anyone. However, experienced programmers can make up for their relative lack of ability to absorb massive amounts of text by memorizing the relevant parts (due to repeat use), and also by searching on Stack Overflow1 or reading the relevant documentation on the fly.

When Karpathy talks about LLMs being good at “boilerplate,” this is exactly what he means. LLMs excel at copying basic setup code from the documentation or from introductory tutorials. But LLMs can go beyond just boilerplate. They are definitely able to string API calls together, or to take the logic for a common problem and adapt it to a different programming language, or a different library, or even a somewhat modified use case. To people with little programming experience, this can appear magical, and it can convince them that an LLM can program anything a user may want. And to experienced programmers, this can save huge amounts of time and effort, in particular when working with a language or library or codebase they are not that familiar with.

But, as useful as this skill is, there comes a time in any programming project where deep conceptual thought is more important. Sometimes you do need to develop a novel algorithm that solves a tricky problem. Or you have to hunt down that weird bug that somehow, for no obvious reason, seems to involve three unrelated components in a large software project. Or you have to architect a new project and there are complex tradeoffs that need to be balanced carefully to arrive at a working solution. In 2025, no LLM can reliably tackle these types of problems.

Maybe eventually LLMs or some other form of AI will achieve proficiency in both skillsets. At that point, AI will be able to program truly autonomously. But we are not there today. Nevertheless, LLMs can be are tremendously useful. They just need to be understood as a more sophisticated version of Stack Overflow, not as an autonomous, junior software developer.

I have had a personal experience recently where I was lacking exactly the knowledge that LLMs can provide. As a consequence, I got huge time savings and increased efficiency out of LLM use.2 It was in the context of the graduate class I am teaching this fall, about AI models in molecular biology. The class covers both (i) the conceptual underpinnings of widely used models and (ii) practical, hands-on experience with building, training, and modifying various AI models, as well as analyzing and visualizing model outputs. I know a lot conceptually about how AI models work. I can explain attention and feed-forward layers and linear projections and activation functions till the cows come home. But I never actually code myself in pytorch.3 And similarly, I know a lot about data analysis and data visualization, but I only have experience doing these kinds of things in R, not in python.4

So with my deep conceptual knowledge about how things work in principle and complete ignorance about how any of this is done in practice, I’d ordinarily have to buckle down, read a ton of documentation and tutorials, and then painstakingly put together my demonstrations and hands-on experiences. It could easily take me two full work days for every one hour of practical in-class material. However, LLMs are exceptional at writing little code examples for a class. All I had to do was ask the AI for code that did what I wanted to do, and the AI would generally deliver useful results within one or sometimes a few tries. You can see an example of the type of prompts I would use here. This made preparing my in-class materials so much simpler and faster. I read every line of code the AI produced and I verified it did what I wanted it to do, but I didn’t have to also read hundreds of pages of documentation to find the exact function calls that would solve my specific problems.

Also, I had various existing code examples that were using pandas and matplotlib and I think both libraries have major conceptual flaws. I didn’t want to teach these libraries. So I needed to convert all these code examples into polars and plotnine. This is a perfect application area for LLMs. Paste the existing pandas/matplotlib code into the prompt box and ask the LLM to translate to polars/plotnine and it’ll zero-shot the answer every time.

Results were a bit more mixed when it came to fixing bugs. For simple bugs, things often worked out very well. I just pasted the error message into the prompt box and the model corrected the code. Typical use cases were situations where the model had hallucinated an API call or a function parameter or a return value, and when it saw the error message it recognized the problem and often came up with the right way to fix the issue. But sometimes this process could go haywire. Just the other day I asked for a fairly simple (I thought) function that could load two protein structures and align them. And the model just couldn’t figure out how to correctly call the superimpose() function from the biotite package. We went through six or seven iterations where the model would give me code, the code wouldn’t run, I’d paste in the error message, the model would respond with new code, which again wouldn’t run, and so on. At some point it felt like we were going in circles, where I got the exact same error messages I had seen in earlier iterations. Eventually, finally, we solved the issue, and arrived at a simple ten lines of working code. But the process felt painful, and in this particular case I suspect that if I had just read the documentation and coded this by hand it would have been faster.

This last example shows how quickly I reached the limits of what even state-of-the-art coding models can do today. Things work great when the task consists of reproducing or slightly modifying existing code examples, but when things go wrong and we need to find a subtle bug the models clearly don’t think. They end up flailing around like a beginner programmer, just trying things out until hopefully something works. In those moments it doesn’t feel like there’s a deep intellect on the other side that is carefully reasoning through the problem and systematically homing in on the root cause of the bug. This task is still on the human user. And more generally, it’s on the human user to realize when the model has gotten stuck, is going in circles, is hallucinating, or otherwise is no longer making useful suggestions.

I believe programming is a niche where LLMs can find product–market fit exactly because so much of programming is reading the documentation and tutorials and code examples. It is an application domain where for specific tasks LLMs are definitely better than humans, and therefore humans who know how to use LLMs appropriately in this context can derive great value. However, I think it is dangerous to get bamboozled by an LLM’s ability to spit out massive amounts of lightly transformed example code and think the model can reason deeply about complex algorithmic or architectural issues. A human who could write straightforward code examples at the speed of an LLM would likely be a superstar programmer, with the associated other qualities superstar programmers have, but LLMs work differently. They don’t have those other qualities. They can generate code, but they can’t program.

More from Genes, Minds, Machines

Half of my subscribers are from the US, and the other half from the rest of the world, with the UK, India, Germany, and Canada being the leading countries outside the US.1

Inside the US, it may be expected that California, New York, and Texas provide the largest subscriber base. They are three of the four most populous states. The remaining state in the top four is Florida. For some reason, people in Florida are not that interested in my posts.2

I was surprised to see Massachusetts and Maryland rank highly in subscriber numbers. They are only the 16th and 18th most populous states, respectively. I suspect subscriber numbers in these states are driven by the large number of people working in higher ed and/or biological research.

Here are the three most popular posts since creation of this blog:

• PhD-level intelligence or the graduate student from hell

• No, AlphaFold has not completely solved protein folding

• We still can’t predict much of anything in biology

If you subscribed for one of these posts, I’d like to express that I’m not very good at writing the same types of articles over and over. Going forward, I will likely write about different topics, unless I have a new point I want to make about a topic I’ve already covered. In general, I’m writing about whatever captures my attention at the moment.

Now I’d like to invite you to provide feedback in the comments. What do you like so far? What could I improve? You’re also welcome to ask me anything. I’ll do my best to answer all questions in the comments.

Let’s dissect this statement carefully. “If the results depend on the random seed you have a bigger problem.” I agree. But here’s the issue. How do you know? If you always use the same random seed, you’ll not realize your results depend on the random seed, because you’ll always get the same results.

A trained statistician might say, “That’s silly, why would anybody do this?” but that’s exactly my point. People who unquestioningly set their random seed to 42 may mess up their analyses in other ways. A random seed of 42 is the brown M&Ms of machine learning and computational modeling.1 The intersection between people who always use random number generators appropriately and those who routinely set their random seed to 42 is extremely small.2

Let’s imagine this conversation between John, a student, and Martin, an experienced machine-learning expert.

“Hey Martin, I’ve run my model five times. I get an accuracy of 98% on the test data every time. My model performs great,” says John.

“That seems too good to be true,” Martin responds. “What’s your performance on the training data?”

“Oh, it’s only about 70%. The model seems to generalize really well.”

Now Martin is getting worried. “You ran the model multiple times, and you got 70% on the training data and 98% on the test data? Did you use the same training–test split each time by any chance?”

“Absolutely not,” John retorts. “I generated a new random training–test split each time, as described in the scikit-learn documentation, using their exact example code.”

Martin is increasingly confused. He hasn’t read the scikit-learn documentation in a while, and so has no idea what it says. He asks John to pull up the documentation.

John pulls out his laptop and says, “Here it is, the example code from scikit-learn:”

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=42)

Martin looks at the example code and says, “But you did change the random state, right?”

“I did not,” says John, now starting to wonder whether he should feel embarrassed about having made a stupid mistake or proud about having thought things through really well. “I didn’t want to give the impression that I cherry-picked my analysis by choosing a specific random seed, so I stuck with the default provided in the documentation. It’s from the Hitchhiker’s Guide. Many people use it.”

If you think this dialog is completely unrealistic then I’m sorry, you need to spend less time with your computer and more time with people.3 Random number generation is an obscure technical topic that most people don’t know much about. Even people who routinely do data analysis or machine learning are not necessarily well informed about how exactly random number generation works. That’s why I wrote my previous post in the first place.

Similarly, I gave my recommendation of not explicitly setting a seed at all because I know how people operate. Yes, this choice sacrifices some reproducibility, and doing something like generating a true random seed and writing it into a log file would be better, but all of this is additional mental overhead that for a good fraction of people will simply be too much. Anybody who has taught students knows that if you provide example code with a seed, some fraction of people will use your code as written and not change the seed. It doesn’t matter how often you say “change the seed.” If your code contains a seed, people will end up using that exact seed, every time.

There’s one more issue. In particular when you’re coding with scikit-learn, you need to set random seeds all over the place. Every single function that has random behavior has its own separate random number generator. So you can quickly face the situation where you need many random seeds. Want to do a train/test split? Please provide a random seed. Want to do a t-SNE? Please provide a random seed. Want to do a PCA? Please provide a random seed. Want to fit a random forest model? Please provide a random seed. You will quickly run into decision fatigue where you won’t have the energy to come up with new random seeds everywhere. You could set up an elaborate scheme where you have a master random number generator which you use to generate random seeds for each step of your analysis, but come on, nobody is going to do this. So I still think it is better to get into the habit of not setting a random seed at all and instead relying on the system random noise the library uses by default.

Let me end with this post on BlueSky by Ehud Karavani. It captures the right attitude. This is what you should be doing.

I cannot emphasize how bad of a choice this is. 42 was a joke guys. Don’t use 42. Ever. If your random seed is 42 I will come to your office and set your computer on fire.1

But is this actually a problem? Do people really use 42 that commonly as their random seed? Yes, absolutely. Google “random seed” or “random_state” and the number 42 will pop up among your top search hits. And people may explain where the number 42 comes from (we’ll get to this below), but then they don’t talk much about whether or not this choice is a good idea. In fact, frequently you see statements along the lines of “the random seed is arbitrary, you can use any number you want, so 42 is a fine choice.” This sentence is 100% correct, assuming you’re the only one who uses 42 and you also use it only once in your entire life. Obviously this assumption is not valid. But I rarely see people point this out.

For example, the documentation for the Python machine-learning framework scikit-learn contains a lot of material about random states and various options of controlling them. Everything the documentation says is technically correct, and yet it never discourages you from using 42 as the seed. In fact, the glossary contains this gem:

Popular integer random seeds are 0 and 42.

(I hope I won’t have to explain why 0 is just as bad a choice as 42.)

The number 42 also shows up throughout the documentation, such as in code examples for the train_test_split() function. Using a fixed random seed when splitting data into training and test sets is uniquely bad, as you’re always going to be sampling the same split when you’re re-training your classifier.

But it gets worse. LLMs have learned about 42 and will happily put it into the code they generate. Here is some zero-shot data-analysis code I recently generated. And there it is, random_state=42 (“for reproducibility”).

It is not surprising that attentive students, who read the documentation, read the blog posts, read the code generated by LLMs, conclude that a random seed of 42 is a good choice, and maybe even a choice that is superior to other options.

To dig deeper into why 42 is not a good choice, and is in fact uniquely bad, we need to look into how random numbers are generated in a computer.

What is a random seed?

In modern computing, we need randomness everywhere. If you’re doing machine learning and you need to subdivide your data into training and test sets, that requires a source of randomness. If you’re writing a computer game and you want your NPCs2 to show somewhat interesting and unpredictable behavior, you need randomness. If you’re simulating a physical system, you need randomness. The problem is that it’s rather complicated to generate true random numbers. The sources of true randomness we have available (for example from thermal fluctuations in specific electronics components) are nowhere fast or cheap enough to generate random numbers at the scale needed in modern computing environments.

The solution computer scientists have come up with is the pseudo-random number generator (PRNG). A PRNG is a mathematical algorithm that produces sequences of numbers statistically indistinguishable from random. Importantly, a PRNG will always produce the exact same sequence of numbers when run from the same starting point. The numbers aren’t random at all! But they look random.

So what is the random seed? It is a number that defines the initial state of the PRNG. How exactly we get from the seed to the initial state can be complicated, but the details don’t matter here. What matters is the same seed will always give you the exact same sequence of random numbers.

A second important concept to be aware of is the period length of a PRNG. The period length is the number of random values a PRNG can generate before it starts repeating. All PRNGs repeat eventually. Therefore, it is critical that your PRNG has a period length large enough that it never causes you any trouble. Let’s say you write a large numerical simulation (maybe you’re simulating the weather, or the early universe, or all the atoms in a cell) where you need trillions or more of random numbers. You wouldn’t want the random numbers to repeat during any of your simulation runs. So you need a PRNG with a period length well in excess of the maximum number of random values you may ever need.

One of the most widely used PRNGs is the Mersenne twister. It has a period length of over 10⁶⁰⁰⁰. (The exact value is 2¹⁹⁹³⁷ − 1.) This is an unimaginably large number. To give you a sense of how large it is, for comparison, there are approximately 10⁸⁰ atoms in the universe. This is tiny compared to 10⁶⁰⁰⁰. The period of the Mersenne twister has space for entire universes for every single atom in the universe, and then some. In fact, you could create an entire universe for every atom, and then create another entire universe for every atom in every of the universes you have created, and keep nesting 75 times, and still you wouldn’t run out of room in the period of the Mersenne twister. If you used the Mersenne twister to create nested universes 75 times deep, all these universes inside universes inside other universes would be different from each other.

What is a good choice for your random seed?

As I wrote above: The random seed is arbitrary. You can pick any seed you want. There are no better or worse seeds. (Unless you have a bad PRNG, but let’s ignore this complication.) In principle we could stop here. But in practice it’s a little more complicated.

While the seed is arbitrary, you don’t ever want to reuse a seed. The point of a PRNG is that its output is statistically indistinguishable from random. That’s going to be the case if you’re using a different seed every time. But if you’re reusing seeds, suddenly you have hidden correlation structures. And you may not even be aware of them.

The consequences of reusing random seeds could be benign or disastrous. It depends on the specifics of the situation. Let’s say you’re doing machine learning, and you’re using the train-test splitting code I quoted above, with a fixed random seed. In this case, you’re always splitting the data in exactly the same way. If you’re running this code five times, you’re not actually getting five independent splits, you’re getting the same split five times. At a minimum, that’s going to be wildly underestimating the variance in the performance of your fitted model. And worse consequences are possible if you’re unlucky.

If you’re following so far, and you’re starting to see that reusing random seeds can be bad, you may also realize that double-digit random seeds are bad. There are only 90 different options. If you’re in any way regularly working with random processes, you’ll easily need 90 different options in just a few days.

So, let’s go wild. Let’s use an 8-digit random seed.3 Surely that’ll give us sufficiently many different possibilities for a lifetime. Well, once you ponder it a bit, you’ll see that this doesn’t even give us a separate random sequence for every person in the United States. (The US population is approximately 340 million.) If they were all doing data science, splitting data into training and test, many of them would be using the exact same “random” splits.

The space you can possibly explore with even quite a long random seed is tiny compared to the total number of sequences you would want to have available to you, and which a PRNG such as the Mersenne Twister would certainly support.4 And the random seed 42 is uniquely bad precisely because everybody is using it. The Mersenne Twister has a state space large enough for universes within universes, but every data scientist in the entire world is using the same 10,000 “random” numbers that you get when starting with seed 42.

So we’re all on the same page, here are the first 50 of the “official” random numbers. They have been used millions of times. I encourage you to verify you get the same numbers on your computer.

>>> import random
>>> random.seed(42)
>>> [random.random() for i in range(50)]
[0.6394267984578837, 0.025010755222666936, 0.27502931836911926, 0.22321073814882275, 0.7364712141640124, 0.6766994874229113, 0.8921795677048454, 0.08693883262941615, 0.4219218196852704, 0.029797219438070344, 0.21863797480360336, 0.5053552881033624, 0.026535969683863625, 0.1988376506866485, 0.6498844377795232, 0.5449414806032167, 0.2204406220406967, 0.5892656838759087, 0.8094304566778266, 0.006498759678061017, 0.8058192518328079, 0.6981393949882269, 0.3402505165179919, 0.15547949981178155, 0.9572130722067812, 0.33659454511262676, 0.09274584338014791, 0.09671637683346401, 0.8474943663474598, 0.6037260313668911, 0.8071282732743802, 0.7297317866938179, 0.5362280914547007, 0.9731157639793706, 0.3785343772083535, 0.552040631273227, 0.8294046642529949, 0.6185197523642461, 0.8617069003107772, 0.577352145256762, 0.7045718362149235, 0.045824383655662215, 0.22789827565154686, 0.28938796360210717, 0.0797919769236275, 0.23279088636103018, 0.10100142940972912, 0.2779736031100921, 0.6356844442644002, 0.36483217897008424]

Should you explicitly set your random seed?

Why choose a specific random seed at all? Is this actually a good idea? In general, I think the answer is no. In my opinion, you’re typically better off using a random5 random seed and sampling a broader space of possibilities than picking your own random seed and risking that you’re drawing invalid conclusions from your analysis. However, there are of course specific situations in which picking a random seed is appropriate, or even required. Let’s discuss those.

Most importantly, in simulation studies, it can be helpful to start all simulations with a different but defined random seed, so that every single simulation run can be reproduced if necessary. In these types of situations, a good strategy is to take some arbitrary large integer number, say 9427385,6 and then add to it the number of the replicate you’re running. So, simulation i would have seed 9427385 + i. This requires a bit of work to get right, as you should never reuse any random seeds among simulations, but it can be useful for tracking down weird behaviors that may occur only occasionally.

Related to this point, when you’re coding a complex stochastic simulation, you may encounter bugs that happen only very rarely. Your simulation may work just fine most of the time, but every few thousand runs or so it crashes. This type of bug can be difficult to investigate. The first step is usually to identify a random seeds that reliably triggers the bug. If you can find a seed that triggers it early in the simulation run that’s even better.

Another scenario in which you might want to use defined random seeds is when you have random choices that you explicitly want to reuse multiple times. For example, if you’re doing machine learning, and you’re comparing two different models, you may want to fit them to the exact same collection of training/test splits. In this situation, you could, for example, pick ten random seeds, use each to generate one training/test split, and fit each model to each of the ten splits.

Finally, when making visualizations that contain random scatter or other elements that are randomly chosen, it can be helpful to play around with the random seed until the scatter looks pleasing. When I wrote my book on data visualization I used this technique quite frequently, for example in this chapter.

Where does 42 come from anyways?

Now that you know everything there is to know about random seeds, let’s go back to the number 42. Where does it come from? Why do people use this number in particular? The culprit is Douglas Adams’ The Hitchhiker’s Guide to the Galaxy, a humorous, quirky science fiction novel. The book was very popular in the 1980s and 1990s, was adapted several times for radio and TV, and reached broad audiences around the world. If you’ve never read it or seen any of the adaptations, I would encourage you to check it out. Just be prepared for rather strange humor.7

In the book, we learn about some hyper-intelligent, pan-dimensional beings that built a massive computer called Deep Thought, with the specific purpose to discover the answer to “the ultimate question of life, the universe, and everything.” After several million years of computing, Deep Thought reveals that the answer is 42. When the beings don’t understand what to do with this answer, Deep Thought tells them that to understand the answer they have to figure out what exactly the question is.8 You can see a film adaptation of this scene here.

The number 42 has since turned into a meme. When somebody asks an extremely broad question, or a question that goes after deep philosophical topics such as the meaning of life, people like to respond with 42. This meme is so popular it has its own Wikipedia article. This is all very funny,9 but none of this makes 42 a good random seed.

Updates

This post generated a number of good responses. I will collect the most helpful or interesting ones here.

First, Nick Bailey pointed out a good solution to the randomness/reproducibility conundrum (where reusing seeds ruins randomness but using true random starting points ruins reproducibility): Generate a genuinely random random seed when you start up your code, write it into a log file, and then seed your random number generator. This gives you the best of both worlds.

A somewhat similar idea: Use the current date as random seed. This at a minimum gives you a different seed each day, while also avoiding the problem of appearing to have fished for the seed that gives you desired results. The devil is in the details though of how exactly you convert the date into a seed. See this post by Stephen Turner and the subsequent replies:

Thanks to Colin Gillespie over on BlueSky, I have learned how to do code searches on GitHub. So now I can report that, as of this writing, there are 496k cases of random_state=42 on GitHub.

Finally, an issue to be aware of if you’re using Matlab: It uses a fixed random seed every time, so the generated random numbers are always the same in a fresh Matlab session:

You can read more about this on the Matlab blog.

To provide an alternative perspective, here I’m re-publishing a lightly edited version of a blog post from 2013, about how most graduate students propose to do too much and need to be reined in in their ambition. The post is primarily about the PhD thesis proposal, where students need to present a plan for their PhD work to a panel of professors. However, much of its content applies more broadly. Even after having successfully defended a thesis proposal many graduate students want to do too much.

I cannot remember ever having seen a graduate student present a PhD thesis proposal and be criticized for lack of ambition. It never happens. Even the weakest students—especially the weakest students—present proposals that are overly ambitious and that won’t ever get done, and certainly not in the 3–4 years remaining until graduation. In fact, in my experience it is exceedingly rare that a student presents a reasonable proposal, one that is actually doable during the remainder of their time in graduate school. Usually, those only happen when students “forget” to have their qualifying exams and end up presenting their “proposal” six months before the intended graduation date. In those cases, the students know that they won’t accomplish much new between proposal day and defense day, and therefore they present a proposal that consists entirely of completed work.

In biology PhD programs in the US, most professors expect graduate students to complete about three projects, corresponding to the magical three specific aims in a typical grant proposal.1 It follows that a graduate student who is defending their proposal, 2–3 years into their program, should have one project completed, one well under way, and one in the early planning stages. Students doing complicated experimental work might have progressed less, but at a minimum they should have one project well under way when they defend their thesis proposal. This leads to a pretty good rule of thumb for the amount of work the proposal should encompass: Aim 1 should be the work that is in the bag, and Aims 2 and 3 together should not require more than twice the amount of work already accomplished.

I rarely see PhD proposals that meet this rule of thumb. Instead, the already completed work is frequently only a small component of the proposed Aim 1, which by itself is going to take another two years to complete. Proposed Aim 2 will need four years on top of that, and Aim 3 another ten. Many graduate students propose to carry out a lifetime of research during their graduate studies.

I don’t quite know why PhD proposals tend to be overly ambitious. Maybe it’s youthful optimism or naiveté. I suspect, though, that there is a component of worry, the eternal graduate student concern of not being sufficiently productive, of not doing enough. Ironically, this concern often causes students to overlook the successes that are within reach and instead try to reach for the stars. In general, doing a successful thesis is a fine balancing act between being overly ambitious and playing it too safe, a topic for another post. However, there’s a difference between an actual PhD thesis and a thesis proposal: The ideal thesis will contain some exciting, risky work, but for the proposal most professors want to see a plan that is doable, not one that might be doable if the stars align correctly. As a smart graduate student, you have two alternative plans, one safe and one daring, you work on both of them at the same time, and you only present the safe one during the committee meeting.2

A second, related issue I frequently notice is that students display poor judgment in how much work they can realistically accomplish in their remaining time. Estimates are consistently too optimistic. If you are in year three, and you have completed 50% of your first project, it is unlikely that you’ll complete this and two entirely different projects in the remaining 2–3 years of your PhD. Further, unless you’re a paper-writing machine, it’s unlikely that you can write a paper in less than three months.3 So, if you still have three manuscripts to complete, plus a thesis, the writing time alone is going to be about a year. If you’re already halfway into year three, you’ll have about another 18 months of actual research work you can do, because the other 12 you’ll spend writing. (Of course I’d recommend that you don’t wait all the way till the end before you start writing, but the math comes out the same.) My personal rule of thumb is things take about three times longer than what students estimate. So if a student says a particular project needs another three months in the lab plus a month to be written up, I expect that project to be done around the same time next year.

In conclusion, when you prepare your thesis proposal, realistically assess how much work you can complete during the remainder of your graduate years. Don’t assume that your productivity will double or triple over the next two years, because it won’t. Budget at least three months for every paper you have to write, and triple the time you think it takes to complete the remaining lab work. If you have papers in review, consider that responding to reviewer comments and revising a paper frequently takes another two to three months, during which nothing else gets done. If you end up with a plan that will require another five years of work or more, then you’ll have to change your aims. See whether your current Aim 1 can be broken down into reasonable sub-aims which can be considered the separate chapters of your thesis. It’s quite common for me to conclude a PhD proposal defense by telling the student it’d be best to scrap Aims 2 and 3 altogether and instead expand Aim 1 into the entire thesis. If you come to this realization before the committee meeting, I won’t have to tell you so during, and everybody is happier.

It’s very tempting, in particular to the tech crowd, to look at this AI success in programming and extrapolate to other application areas. One popular area of extrapolation is biology. If we can teach an AI programming by feeding it millions of examples of code snippets, so the thinking goes, surely we can also teach it biology by feeding it millions of examples of biological data. And yes, to some extent this works. AlphaFold is pretty good.2 But, in practice, what happens more often than not, is not what I would call the AlphaFold experience.

This is my usual experience: I read about some new computational method that appears to work exceptionally well, I get excited because it’s exactly what I need for one of my projects, I try the method, and results are disappointing. Most of the time things don’t work, or at least not as well as expected. I have seen this play out so many times my default assumption is nothing is going to work. And anything that does actually work is a bit of a miracle. The only successful strategy is volume. If you’re trying sufficiently many things, some do in fact work, and those you can publish.

My latest example is with the software BindCraft, a tool built on top of AlphaFold to design peptide binders. Before I continue, let me emphasize that in my opinion BindCraft is really good. I have the utmost respect for its creators. It’s well documented, it’s easy to install, it does what it claims to do. Without doubt, it’s the best tool for designing peptide binders currently on the market. And yet, even though the authors write that they achieve “de novo protein binder design with experimental success rates of 10–100%,” in our own hands only maybe one out of about a hundred designs actually works. We can design binders with BindCraft, and we can design them more efficiently than using any other available method, but we still have to experimentally test hundreds of designs to get a handful of working inhibitors.3

This story is not unique to AI methods. I’ve been in computational biology for nearly three decades. Nothing has fundamentally changed during this entire time between how papers describe the success of new computational methods and how the methods actually perform in practice when you use them on your own system of interest. I remember, in the early 2000, David Baker was revolutionizing computational protein design with his Rosetta software suite, winning CASP competitions left and right, and writing papers that gave the impression computational protein design was solved. For example, computational design of novel folds was solved by 2003, protein docking was solved by 2003, enzyme design was solved by 2008, and atom-level co-folding of multiple peptide chains was solved by 2009. And yet, here we are, twenty years later, all of these topics are still active areas of research, and if you have any particular system of interest you may find that none of the available methods perform that well.

Now let me be absolutely clear: I’m not accusing anybody of faking data, doing sloppy science, or misrepresenting their results. All the papers I have cited here are examples of outstanding science practiced at the highest level. Instead, I think there are several things going on. First, biology is really difficult. Just because something works in one system doesn’t mean it’ll work in another. Second, there is publication bias. The examples that work get published, the ones that don’t work do not.4 Third, experienced PIs develop an instinctual understanding of which specific problems are amenable to their methodology. They may subconsciously or deliberately choose problems where the likelihood of success is high. I’m sure if you went to David Baker with a protein-design problem he could tell you right away whether you’re likely going to succeed or not and which software and parameter settings would maximize your chances. But, you’re not David Baker, you don’t have three decades of experience designing proteins, and so you’ll naively pick the wrong method or attempt a problem that is simply too hard. And the experience will be frustrating and it’ll feel like the available methods don’t perform anywhere near as well as the published papers suggest.

Going back to our experience with BindCraft, we’re starting to see patterns emerge of when it succeeds and when it fails. BindCraft appears to be much better at targeting certain binding pockets than others. You wouldn’t easily find this out from reading the paper, but generate a few thousand designs and you’re starting to get a feel for how the algorithm behaves. Similarly, there are non-obvious aspects to how you prepare your target structure that matter. In general, you have to make hundreds of seemingly inconsequential choices that can increase or decrease your chances of obtaining a successful binder design. Make all the right choices—and sprinkle in a bit of luck—and your design will work. Make a couple of wrong choices and it’ll fail.

I want to emphasize that this post is not meant as a complaint about BindCraft. What I describe is a general phenomenon. For another example, look at my recent post about how protein language models fail on viral data.5 Specifically, look at the graph I included in that post. Even for the non-viral proteins, for at least half the datasets predictions mostly or entirely fail. An R² below 0.5 is not a good prediction.

Why are predictions so poor? How can biology be so difficult? Biology is just physics and chemistry, and in principle we should be able to write down the equations of motion of any biological system and solve them numerically. In practice, however, any realistic biological system is way too large for this approach and we don’t have the required compute. It’s not feasible to rent an entire supercomputer for a year just to calculate the fitness effect of a single mutation. So we have to rely on shortcuts. The shortcuts that work are database lookups and interpolation. AlphaFold is a gigantic lookup and interpolation machine. It uses known structures in concert with covariation derived from multiple sequence alignments to link sequences to likely structures.6 Ask it to predict structures of novel folds or of sequences with no known homologs and its performance often craters. I’ve written previously about the limitations of AlphaFold. And a software such as BindCraft, built on AlphaFold, inherits all of AlphaFold’s limitations. To improve binder design, we first need to improve protein folding predictions.

And up to this point we’ve mostly talked about structure prediction. Move up one level to protein function and things get exponentially more difficult. Any given protein can have hundreds of different functions, when you consider all the different environments in which the protein can be expressed, different interaction partners it may come in contact with, and—for enzymes—different substrates it may act upon. If you wanted to construct a model that could reliably predict effects of mutations, you’d need it to achieve this task for every possible mutation in every single context in which the protein could occur. I hope you can see how big of a task this is, and, for AI approaches, how much data you’d need to collect to cover this enormous space evenly in your training set.

And once you’ve mastered protein function, you’re still only at the base of a gigantic mountain. First, there are molecules beyond proteins that also play important roles, such as RNA, DNA, lipids, glycans, various other small molecules, the list goes on and on. And second, you have to assemble individual molecules and their functions into pathways, and then into cells, and organ systems, and organisms, and from there on into populations and eventually ecosystems. You will encounter new challenges and complications at every new level of organization. You may hope that you can abstract away the lower levels as you move up in the hierarchy, but this will only get you so far. Individual genetic changes can have visible consequences at much higher levels of organization. As but one example, consider genetic variation linked to burrowing behavior in mice. In the end, all of biology is steered by what happens at the molecular level, and you can never quite get rid of the complexity and the unexpected effects or interactions.

I want to close on a more positive note. Even though computational predictions in biology are still fraught with failure, with every passing year science grinds forwards and we’re collectively accumulating knowledge. Our ability to manipulate biological systems is improving one step at a time. For example, we are much better today at designing peptide binders than we were twenty years ago. And at the same time, we still have many decades of work ahead of us. The world of biology is so large that complete mastery remains out of reach. Today we can design a car or an airplane entirely in silico, and when we build it it works on first try. Maybe, in a few decades, we’ll be able to do something comparable with a biological system.

Now I’d like to hear from you in the comments. Do you have similar experiences? Do your modeling or prediction efforts frequently fail? Or do you think I’m too negative and in your hands things generally work? In your mind, what’s the state of computational prediction in biology in 2025?

This post, or at least something like it, was requested by Eurydice.

After the personal statement, the research plan is the next-most important component of a GRFP application. It is where you describe what research you plan to carry out. And you only have two pages to do so. This means you can’t actually write a detailed research plan. What you have to write is an outline of a plan. I think of the research plan as an extended summary page. If you’re familiar with the standard NIH proposal structure,1 think of the Specific Aims page, and just expand it a little bit. There you go. You’ve got your NSF GRFP research plan. (But don’t forget the Broader Impacts section.)

There’s a lot to be said about how to write a research plan, and I can’t possibly fit everything into a single post. So I’ll start by linking to a few other proposal-writing resources, and then I’ll provide suggestions that are specific to the NSF GRFP. First, I suggest you go and read this old blog post of mine about fine-grained sectioning in proposals. Then, you may find this blog post about the critical need helpful. Finally, here is a handout from a class I taught some years ago. The handout has various pointers about writing grant proposals. Most importantly though, on the second page, it has a sentence-by-sentence outline of a proposal summary page. Just follow the outline and your research plan will write itself.2

Write for a broad audience

A common mistake I frequently see students make is to assume everybody knows about their research area. Students may have spent years in a lab where everybody talks about the same problems day-in day-out and now they think anybody who is broadly in the same field will be intimately familiar with the specific questions and issues they deal with every day. Nothing could be further from the truth. It’s best you assume nobody knows anything about your research. Yes, the reviewers are all professors in your area. And no, they have never heard of the system you’re studying.

To give a concrete example, in biology, students working on computation frequently misjudge what computational methods the average experimental biologist is familiar with. As a consequence, they write proposals that most reviewers can’t understand. And the reverse is also true. Experimental students may assume certain approaches are well known when maybe only ten labs in the world do these particular experiments.

So, do your best to write for a broad audience. But also, be aware you may still misjudge what is and isn’t broadly known. As a simple test to assess how comprehensible your writing is, show your proposal to a fellow student in your cohort but not in your lab. If they don’t understand your research plan chances are the reviewers won’t understand it either.

Stay away from biomedical research

There’s an unwritten rule that the NSF doesn’t like to fund anything that is clearly in NIH territory. The NIH supports research related to human disease. Therefore, the NSF generally stays away from human disease. The reasoning is that the NIH has a budget that is five times bigger than the NSF budget, and yet the NSF is tasked with funding all of science whereas the NIH only funds research into human disease. So that’s one area the NSF doesn’t see the need to spend its scarce resources on.

If you’re a physicist, a chemist, a computer scientist, or an engineer, there is little risk that you’ll accidentally propose something that looks like NIH material. But, if you’re a biologist, this is a realistic concern, since the majority of biological research in the US is funded by the NIH. If you’re an undergraduate or first-year graduate student doing research in a biological lab in the US there is a good chance that your lab is NIH funded and so you’ll just instinctively absorb NIH language and priorities. If this is the case for you, I encourage you to deliberately stay away from NIH language and from human disease.3

Don’t worry about committing to a specific research program

Receiving a GRFP award does not obligate you to do the exact research you have proposed. It’s easy to see why this should be the case: The vast majority of students who are applying are either not yet in graduate school or may be in graduate school but are in a rotation system and haven’t yet chosen a permanent lab. In both of these cases, students may end up in a situation where they literally cannot do the research they have proposed. For example, if you propose to do research on penguins, and then you don’t get into the penguin lab or even the school where the penguin lab is and instead end up in a butterfly lab, you’ll have to do butterfly research. But even if you do end up in the penguin lab, by the time you’re situated and the GRFP has started almost a year will have passed from when you applied and the specific research you suggested may no longer be relevant, or interesting, or the most impactful way for you to spend your time. Or, you end up having a long conversation with your advisor and they explain to you why maybe you were a bit naive when you wrote your research plan and some parts of it are infeasible, too costly, or uninteresting. In all of these cases, the right thing to do is go after the research that makes the most sense, not the exact research you proposed.

What you are writing in the research plan is an outline of what a reasonable PhD project could look like for you. You’re not writing a contract in blood that you’ll be tied to for the entire three years of GRFP support. Changes in research plans are normal and occur all the time. Changes in advisors are normal and do occur. Even changes in graduate program or university are normal and occur from time to time. You can be awarded an NSF GRFP and still make any of these changes in your PhD research if you want to.

Write a research plan about the research you know best

It is a good idea to be somewhat strategic about the research you propose, in particular if you haven’t yet joined a permanent lab. Write about the research you are most familiar with, even if it’s not necessarily exactly the project you want to do for your PhD.

As an example, let’s assume there are two labs you might want to join, one you’d be very excited about but it’s a little further thematically from research you have already done, and another one that is more aligned with your prior research experiences but it’s less exciting to you. In terms of the proposal writing, you will likely be better off writing a proposal related to the second option, even if you’re confident you’d join the first lab if both options were available to you. In proposal writing, lack of specificity or detail is the kiss of death. It’s unlikely you’ll be able to write a strong proposal about research you haven’t done previously and have no experience with.

Structure your research plan into well defined objectives

Your research plan needs an overarching theme or goal or research question, and then two or three well defined objectives you will pursue to achieve the overarching goal. The right number of objectives is critical. Three is ideal, and two is fine. More than three is too ambitious, and less than two4 means you haven’t actually developed a proper plan.

Each objective should have a descriptive title in bold and then about a paragraph of text describing what you will do. It can also help to add a hypothesis after the title and before the paragraph of text. To get an idea of what I mean, take a look at my old blog post on fine-grained sectioning in research proposals. You can also look at the first page of this old grant proposal I’ve sometimes used as an example in classes I’ve taught. But, note that both the blog post and the grant use the term “Aim.” Write “Objective” instead. Remember, you’re not applying to the NIH.

Address the broader impacts and the intellectual merit

In your research plan, you need a separate section called “Broader Impacts.” Realistically though, you don’t have sufficient space in your research plan to write much about broader impacts. My recommendation is to limit this section to a single paragraph, in which you summarize the broader impacts plan that you have spelled out in more detail in your personal statement. See my post on the personal statement for details.

And you also need a section on “Intellectual Merit.” My post on the personal statement addresses this also, so I’m not going to repeat myself here. Go read the prior post if you haven’t done so yet.

The intended audience for my post is undergraduates and first-year graduate students in the US applying for an NSF GRFP. If this is not you, there’s no need to continue reading. However, some parts of the post touch upon general thoughts about personal statements and grant applications, so even if you’re not in the US or not a student you may find some of the remainder useful.

The instructions for this year’s application are available on the NSF website. If you’re planning to apply, read these instructions very carefully, from beginning to end. No matter what I say here, if at any point my advice disagrees with what the NSF writes the NSF is correct. It is ultimately your responsibility to adhere to all the requirements spelled out in the application guidelines.1

When you read the guidelines, you will see that the two main documents you have to prepare are a personal statement and a research plan. Of those two, the personal statement is the more important one, for two reasons. First, it is substantially longer—three pages instead of two. That in itself tells you it is more important. There is more space to say something interesting or to make a fool of yourself. Second, I believe2 that reviewers put more stock into the personal statement. Here is my reasoning: Most students receive more help and feedback for their research plan than for their personal statement. As a consequence, many of the submitted research plans are pretty good, and reviewers can’t use them to separate the absolute best students from the only very good students. And also, if a research plan is truly excellent, reviewers will worry that the ideas came from the student’s supervisor rather than from the student themself. So, an excellent research plan with a mediocre personal statement suggests the student got a lot of help but isn’t that strong when on their own. By contrast, an Ok research plan with a strong personal statement suggests the student may be able to punch above their weight but didn’t receive as much coaching when preparing the application or may not yet have a lot of research experience. Therefore, all else being equal, I expect most reviewers will weigh the personal statement more than the research plan. Of course, the absolute best students will have both a strong research plan and a strong personal statement, and you should try to achieve this level of excellence.

Share

Now that you know why the personal statement is important, let’s discuss specific strategies for making it good.

Approach the personal statement with the right attitude

Let’s begin with the overall attitude with which to approach your personal statement. Your application will be reviewed by established scientists. What they’re looking for is a peer, somebody they can treat as a colleague from day one—not a student who will need a lot of hand-holding.3 So your personal statement needs to convey you’re this type of student. Think about what a scientist would tell other scientists about themselves. Would they start their personal statement with something like the following?

As a teenager, I was fascinated with the natural world, and in particular with insects. I would observe ants as they were building nests, and I would collect butterflies. My high school science teacher Ms. Smith was a beekeeper and one spring she showed us how she maintained the hive and harvested honey. It made me wonder why the bees always stayed with the hive instead of flying away. And also, when bees leave to forage, how do they manage to find their way back home?

The answer is most likely they would not. They would not because they all have a similar origin story. In high school, they were all weird nerds nobody wanted to hang out with and who could geek out for hours over obscure topics.4 None of your experiences in high school differentiate you from the other applicants with those same experiences, and those experiences are also not relevant for your likely success or failure in graduate school.

An established scientist would simply lead with their science. They would lay out a grand vision of what they’re trying to accomplish, and then describe in more detail what they have done so far along this route, what successes they have had, what stumbling blocks they may have encountered, and so on. For example, if I had to write a personal statement for the NSF GRFP, I might write something like the following. (But note that for the NSF GRFP, you probably want to stay away from an emphasis on biomedical research. I emphasized biomedical research here so if you’re working in an area similar to mine you aren’t tempted to just copy my text and paste it into your statement.)

I am a computational biologist and data scientist. I use mathematical modeling, computer simulation, and machine learning to address questions of biomedical relevance. Much of my work focuses on disease-causing agents in humans, and my overarching goal is to use computational tools to speed up the discovery of disease mechanisms and design of therapeutic agents. I currently have active research projects developing antibacterial agents, identifying novel molecular or genetic targets for antiviral treatments, and designing inhibitors of gene-editing enzymes.

You may not feel comfortable presenting yourself in this way, as an established scientist who has already accomplished various things and knows what they want to do going forward. You may feel that this is not you, at least not yet. But I can assure you that there will be applicants who have this down, so find your inner scientist and try to bring it to the fore in your own application. Importantly, note that how professional you come across is not determined by how much you brag about yourself (you shouldn’t) but by your selection of topics. Write about your research. Don’t write about your experiences in high school science class.

Provide your readers with an abstract

Write the first paragraph of your personal statement as an abstract. Write six to eight sentences that cover all relevant components: Who you are, what you currently do, what your research is about, what your future goals are. This is a general writing trick when preparing any sort of proposal or application. Even if (or in particular if) the instructions don’t ask for an abstract, write one. But don’t label it “abstract.” Just turn the first paragraph of your document into a summary of the whole.

To understand why I’m giving this advice, put yourself into the mind of your reviewers. They have ten or more applications to review, and they don’t want it to take all weekend.5 So they would like to figure out quickly which applications to rank at the top and which at the bottom. If you can give them a single paragraph that allows them to envisage your entire application, you’re already on their good side. The worst thing for a reviewer is when they’re reading page after page and have no idea what any of this is about. You won’t believe how many personal statements I’ve seen where I’m two pages in, we’ve finally made it out of high school and into freshman year in college, and I still have no idea where any of this is going and what the applicant wants to tell me.

Your statement doesn’t have to be in chronological order

Continuing on from the abstract, you want to lay out your specific argument for who you are and why you deserve a fellowship. This argument does not have to be arranged in chronological order, starting with grade school and ending in the present. While the majority of personal statements are arranged in chronological order, this arrangement often makes them long-winded and tedious to read.

For example, when you describe your prior research experiences, don’t arrange them in the order in which they occurred unless there’s a good reason to do so. Instead, organize the material such that it most supports your overarching story. If that results in a chronological ordering, fine. But if not, also fine.

Think about the various research experiences you’ve had, and for every one identify one particular achievement, insight, or observation you want to highlight. Then arrange them such that the sequence of events maximally engages your audience. For example, maybe you first present a success you’ve had, then you present a failure where you’ve learned an important lesson, and then you talk about how the lesson you learned will shape your research approach going forward.

Separately address intellectual merit and broader impacts

You are required to write separate sections for “intellectual merit” and “broader impacts” in both the personal statement and the research plan. This is stated in bold in the application preparation instructions. If you don’t have those sections your application will not get reviewed. As an undergrad or first-year graduate student, this may be the first time you’re seeing these two terms, and you may have no idea what they mean. They are NSF-specific terms and it’s best to start with the definitions the NSF provides:

Intellectual merit: The potential for the proposed project to advance knowledge and understanding within its own field or across different fields.

Broader impacts: The potential for the proposed project to benefit society and contribute to the achievement of specific, desired societal outcomes.

Intellectual merit is the significance of your project, the why. Why should your project be done, and why should anybody care? Ideally you have some knowledge gap or unsolved scientific problem that your research is going to address, and closing the gap or solving the problem is the intellectual merit of the project.

Broader impacts on first glance may seem similar. Isn’t it also the why of your project, only a bit broader? This is a common misreading of what broader impacts are. Experienced NSF investigators know that broader impacts will generally describe a distinct activity that is carried out, separately from the actual research, and that often involves some form of education or outreach (see also my next section). The NSF has a page dedicated to explaining broader impacts, though even that page has to be taken with a grain of salt. What the NSF says broader impacts are and what reviewers score highly on the broader impacts category is not necessarily perfectly aligned. In case of doubt, ask somebody with extensive experience of receiving NSF funding and sitting on NSF panels to evaluate your broader impacts component.

Develop a broader impacts activity

A competitive application will typically propose a specific activity that results in broader impacts. It is not sufficient to just talk about how the proposed research is important and may have far-reaching societal consequences. Instead, you need to be actively doing something that is not research and that can be clearly described as broader impacts.6

Here are some examples of what broader impacts activities could be: (1) You’ll be organizing a club for students with similar research interests to yours. Part of the club’s goals will be to help interested undergraduates find their way into your research field. (2) You’ll be reaching out to high schools and visit classrooms and tell students about your research or research in general. (3) You’ll be giving public lectures. Or, you’ll be organizing a public lecture series where you connect an interested public with accomplished scientists. (4) You’ll be developing or maintaining some widely used scientific or educational software, or develop some other resource that will benefit society broadly.7 In the past, activities that would increase diversity of the student body or help underrepresented minorities were highly valued in the review process, but in 2025 I would stay away from any activities that require you to write these words to describe them.

Whatever you decide to propose, make sure it is credible and feasible. You increase credibility by providing specific details. For example, simply saying “I will reach out to local high schools” is not very credible. If instead you say something like the following the added detail conveys that you’ve thought this through and know what you’ll be doing: “I will work with high schools XXX, YYY, and ZZZ. I have already talked to Mr. Smith who teaches science classes at XXX and Ms. Miller who teaches at YYY. Ms. Miller will also connect me with teachers at ZZZ.” If you’ve already done the same type of activity in the past, even better. Include some concrete examples in your statement.

In principle, you can describe your broader impacts activity either in the research plan or in the personal statement. The NSF instructions don’t provide guidance on this choice. However, I believe the personal statement is the better option. For one, you have an additional page, and the research plan is going to be tight already. Also, it fits better thematically. Placing the broader impacts activity into the personal statement underlines that it’s separate from the proposed research.

In conclusion

Take your personal statement seriously. Chances are it will make or break your application. Ask mentors or friends to read it and give you feedback. Do they find what you wrote compelling? Are they confused by anything? Do they have concerns that something you’re proposing is not feasible? It is Ok to ask family members who are not academics to give you feedback, but realize they will likely not know what a personal statement should look like and may give bad advice. Their feedback can be helpful in terms of clarity of writing, grammar, sentence structure, and so on, but they’ll most likely not be able to judge whether you’re saying the right things in the right order. If you can have an experienced academic review your personal statement that’s always a better option.

Update: Part II of this series is now available here.

I realize the irony of writing this on a site that is all built on paid subscriptions. It frustrates me, because I want to support writers and I think Substack gets a lot of things right, such as giving writers control over their distribution lists or providing an easy platform for discovery and promotion.2 But Substack is also full of dark patterns.3 Examples include trying to get you to subscribe before even reading a single article; trying to get you to subscribe to other publications as part of the standard onboarding process; not showing you the details of the paid options unless you click on the subscribe button; replacing articles with a subscribe button if you stop reading but don’t close the tab; the list goes on and on. One could teach a college course about dark patterns and exclusively use Substack examples for illustration.4

More importantly, however, I think subscription in itself is a dark pattern. Imagine you went to a store to buy bananas, and they said “sorry, we don’t do one-time sales; you have to sign up for monthly delivery; how many bananas do you need a month?” Or you went to a restaurant to have dinner, and they said “we only serve members; subscribe to the one, two, or four meals a month plan and we’ll seat you right away.” But of course these days all of these things exist as subscription services, because vendors have realized that if they can get you to subscribe you will generally buy more from them than you otherwise would have, and probably more than you needed. As a case in point, consider Amazon. If you try to order anything from them that you might need more than once, such as toilet paper or pet food or cooking oil, they try very hard to sell you a subscription service instead. On occasion I’ve found it quite confusing to figure out how to place a one-time order. I’m sure people have taken out subscriptions on Amazon just because they were tired of fighting over making a one-time purchase.

But Amazon at least gives you the option of a one-time sale. Substack does not. You either subscribe, or there’s nothing here for you to spend your money on. So, if you do like the platform in principle, and if you want to support some of the writers you encounter here, you will end up with an ever growing collection of paid subscriptions.

To counteract this trend, I recently went through all of my paid subscriptions on Substack and cancelled them one by one.5 As I did so, I discovered one I didn’t even know I had. This demonstrates the dark pattern at work. Had I not systematically checked, I would have continued paying for who knows how long.6

I’m not convinced that the dark pattern of trying to trick people into forgetting their subscriptions is worth it. I know that for myself, because I’m so aware of this pattern, I tend to be extremely reluctant to commit to a paid subscription. I’d be much less worried about making a one-time payment, as if I was buying an author’s book or attending a live event.

For this reason, I have decided that going forward, when I want to support an author on Substack, I take out an annual subscription, which I then cancel immediately. This is in effect a one-year non-renewing subscription, and it provides the author with a meaningful level of support. Once the year is up and the subscription ends, if I wish it had continued, I’ll just take out another one-year subscription.

After just a few days of trying this out, I have found that this mindset makes it much easier for me to take out a paid subscription for an author I like. I’m probably willing to do about one yearly subscription per month, given current Substack prices. The feeling of no longer having to worry about all these future recurring payments is rather freeing.

I realize that as I’m doing this and saying this I’m in a way validating Substack’s dark patterns. By developing strategies to work around their machinations, I’m removing pressure on Substack to enact any changes.7 However, I’m under no illusion that whatever I say could have any influence on what Substack does. I’m far too unimportant and invisible. So, what I offer here is akin to a lecture on defense against the dark arts, for the handful of people that may read this. If I can help even a few people feel better about paid subscriptions, that seems a worthwhile achievement to me.

I have seen people argue that Substack should offer one-time payments for specific articles. I understand why Substack doesn’t want to do this. People claim they would buy articles one-off but realistically the volume would be so low that the vast majority of writers would generate virtually no income from this option. However, I do think Substack should offer yearly non-recurring subscription payments.8 Give me the option of buying a one-year subscription without having to worry about renewal. Maybe even give me the option of buying multiple years at once.

But, I don’t have high hopes that Substack will implement non-recurring subscriptions. And therefore, for now, I encourage everybody to just cancel their (annual) subscriptions right after paying for it. There’s nothing wrong with doing this. Nothing bad will happen. You will have access to the paid content for a full year.9 And also, you’re not a bad person if you cancel the same day you subscribe. You paid for a year. Most people don’t do this. And you can pay again. If you continue to like an author chances are you will want to pay again. And if in a year’s time you don’t even remember why you ever paid a certain author, it’s probably for the best the subscription is cancelled. So go ahead, cancel your subscriptions today.

First, just because things are changing doesn’t mean everything is getting worse or changing for the reasons claimed. The article glosses over such complications, for example by focusing on averages: On average, math scores are down, reading scores are down, books contain shorter sentences.1 But the average can go down even as the variance increases and the top performers get better.2 It’s possible that we’re past peak literacy on a broad population basis—since many people now can obtain entertainment without having to read much—and yet civilization as whole may not be in decline and reading can continue to be critically important to a meaningful subset of the population. In fact, modern information technologies, including cell phones and the internet, provide nearly limitless access to information, and people take advantage of it in ways that would never have been possible even twenty years ago.

As a case in point, why are you reading this? Didn’t you get the memo? You should be on TikTok or Instagram or YouTube (but only in the shorts section, please). If reading is declining, how come a site like Substack, which is so centered around long-form and fairly intellectual content, keeps growing while much easier to consume content such as Fox News or CNN is declining?

In the same vein, who would have predicted the rise of the 3 hour podcast? It takes serious focus and effort to sit through one of those. Yet the genre is more popular than ever.

Share

The article also relies heavily on associations, presented as if they necessarily reflected causal relationships. And it has issues with the time scales on which different trends can be seen. In particular, sentence complexity in books has been steadily declining for nearly a century, not something we can attribute solely to cell phones or even the internet.

I could go on about the various statistical flaws and logical fallacies in the article. (Don’t get me started on the regression of vote share versus TikTok interest in Romania, shown towards the end of the article.) But instead, for the remainder of my post here, I want to focus on one specific comment, regarding a study3 about the decline of reading comprehension in English literature students in the US. I quote:

A study of English literature students at American universities found that they were unable to understand the first paragraph of Charles Dickens’s novel Bleak House — a book that was once regularly read by children.

There are so many issues. First, these were students at two “regional Kansas Universities.” So please don’t imagine Harvard or Stanford undergraduates here. Second, the study was done in 2015. That doesn’t align well with the author’s thesis that cell phones and social media are to blame, does it? Third, I feel there’s a major confounding effect here. By using Bleak House, the study did not just assess students’ reading comprehension but also, implicitly, their knowledge and interest in 19th century English culture and way of life.4

In case you’re not familiar with Bleak House, this is the first paragraph of the novel:5

LONDON. Michaelmas term lately over, and the Lord Chancellor sitting in Lincoln’s Inn Hall. Implacable November weather. As much mud in the streets, as if the waters had but newly retired from the face of the earth, and it would not be wonderful to meet a Megalosaurus, forty feet long or so, waddling like an elephantine lizard up Holborn Hill. Smoke lowering down from chimney-pots, making a soft black drizzle with flakes of soot in it as big as full-grown snowflakes—gone into mourning, one might imagine, for the death of the sun. Dogs, undistinguishable in mire. Horses, scarcely better; splashed to their very blinkers. Foot passengers, jostling one another’s umbrellas, in a general infection of ill-temper, and losing their foot-hold at street-corners, where tens of thousands of other foot passengers have been slipping and sliding since the day broke (if this day ever broke), adding new deposits to the crust upon crust of mud, sticking at those points tenaciously to the pavement, and accumulating at compound interest.

I don’t know about you, but for me the first sentence, a mere 13 words, contains 7 that I’d have to look up (“Michaelsmas term”, “Lord Chancellor”, “Lincoln’s Inn Hall”). As a consequence, upon reading the first sentence I immediately lose interest.6 Can I read this text if I have to? Yes. Do I want to? No.

Anyways, I’m sure American college students are as challenged at reading comprehension as the study claims, but nevertheless the study approach bothers me. When we make students read prose describing a world that is entirely alien to them, using numerous words and concepts they cannot relate to, is it so weird that they might not understand, and more importantly, might not be motivated to make a real effort?7 I’d very much prefer an assessment of reading comprehension where the material is more contemporary, so the question of whether students can comprehend the writing is separate from the question of whether they have intimate knowledge about living in 19th century England.

There’s one part of the study that truly irritates me. It recounts how a study participant describes a brief excerpt from Bleak House. A quote of the relevant part of the study follows below. (You can read the study in its entirety here.) In the quote, the excerpt is labeled “Original Text,” the study participant is labeled “Subject,” and the person interviewing the subject is labeled “Facilitator.” The paragraph at the end of the quote is the take-away of the study authors.

Original Text: On such an afternoon, if ever, the Lord High Chancellor ought to be sitting here—as here he is—with a foggy glory round his head, softly fenced in with crimson cloth and curtains, addressed by a large advocate with great whiskers, a little voice, and an interminable brief, and outwardly directing his contemplation to the lantern in the roof, where he can see nothing but fog.

Subject: Describing him in a room with an animal I think? Great whiskers?

Facilitator: [Laughs.]

Subject: A cat?

Note that the subject, who is not accessing any of the concrete details in the passage, finds a subject (the Lord Chancellor) and one recognizable word, [End Page 9] “whiskers,” and concludes that the character is in a room with a cat. At this point, she does not seem to understand what she is reading, and so she links a few words together to form some kind of response.

I find this interpretation of the subject’s response condescending. Assume you know nothing about Bleak House but otherwise have good reading comprehension. Now re-read the quoted text. Is it so weird to read “whiskers” and think about a cat? In particular, imagine you’re a contemporary student, who may have grown up with material such as The Wizard of Oz, Wicked, or Harry Potter, where animals take on major roles in the story development. Really put yourself into the position of the subject. You’re reading a weird book, about a weird world that doesn’t seem anything like your own, but that maybe reminds you of fantasy movies or similar material you have been exposed to. Now you read “addressed by a large advocate with great whiskers” and maybe you go “well, I don’t really understand this, but let’s assume in this world animals can talk and work as lawyers, and so maybe this advocate here is a cat.” I think this is a perfectly reasonable conclusion to draw.8

I’m sure most English majors consider reading something like Bleak House purely as a study assignment. Not something anyone does for fun. And how can you blame them? Just because it’s a classic doesn’t mean it’s enjoyable. Who decided that if you’re interested in the English language you also must like the culture and way of thinking of 19th century England?9

In my opinion, if we want students to read more, we need to introduce them to texts they can actually relate to and enjoy. I don’t know what those would be, that’s outside my area of expertise. But I personally have read plenty of books by different authors and there have been many that grabbed me, and yet I’ve never been able to make it even half-way through one of the 19th century classics. I’m just not interested in the worlds they describe. I can’t blame US high school or college students for feeling similarly.

Evan Dorn writes on LinkedIn:

I suffer from intense ADHD, which wasn’t diagnosed until just a couple of years ago. And even then only because the diagnostic criteria have changed & matured substantially since I was first tested ~2002.

I battle with executive function every minute of every day, and until I understood the cause and received treatment, I lived with decades of self-loathing thinking I was just a failure because I struggled with “simple” tasks that almost everyone takes for granted. In retrospect, it’s a bit of a miracle that I completed any of my degrees, much less my PhD. But compensating mechanisms learned through trial and error got me over the line, even though many of them are quite unpleasant when experienced from the inside.

Today I’ve got a much better understanding of how my own brain works, as well as greatly improved therapeutic help, and both make a huge difference. But I do wish more people had your level of empathy for conditions that include executive dysfunction. In a world that judges people’s moral worth by their productivity – even to the point of whether they deserve to have a decent standard of living – it can be an extremely unpleasant way to live.

One person has reached out to me by email. I’m quoting them here anonymously:

ADHDers in grad school need things not to be boring. You’re right about too big of a task being difficult to start, so coaching them on how to find the first step is good. But the small tasks might sometimes be boring and ADHD brains can’t focus and engage in a boring task. Sometimes boring things have to get done still, so strategies like music, color and body doubling help.

Smart ADHDers are also really afraid of failure. They don’t trust themselves to be good enough and are always right on the edge of failing (in their minds) so they’d rather procrastinate a task than risk failing at it. Forcing them to confront that fear and overcome it is important.

Finally, Tom Devitt comments under the article:

Claus, fellow professor here. I know you’re trying to help, and I’m saying this with all due respect: parts of your post land as demeaning and shaming to the very students you want to support. I suggest a different playbook.

Improving executive function doesn't boil down to “doing what you’re told, but smaller.” For many neurodivergent students (PhD candidates and undergraduates alike), the bottlenecks are time perception, salience, and working memory. Time blindness means the world collapses into two buckets: now and not-now. “Read one paper and write a paragraph” feels trivial, so it slides into not-now until a hard trigger makes it suddenly now, usually too late. Working-memory limits add an object-permanence effect: if the task isn’t literally in front of me—almost always better in analog rather than digital form—it may as well not exist. On top of that, ADHD is an interest-based nervous system: activation turns on with novelty, challenge, degree of interest, or immediate social consequence. That’s why students often jump to the most challenging part first; paradoxically, it’s the one that finally flips the “on” switch.

This is also why “break it into simpler tasks” often backfires. Shrinking scope without changing time, salience, or visibility lowers meaning (“trivial → not-now”), multiplies context switches, and creates shame loops (“I couldn’t even do the small thing”). If you want different results, change the environment, not just the size of the chores.

There’s also a real upside to neurodiversity that I think you’re missing. Many neurodivergent individuals are big-picture thinkers, capable of rapid association, systems-level synthesis, and deep hyperfocus when their level of interest or novelty is high. There are plenty of highly successful, neurodivergent people out there who just have different wiring, which has real advantages in the proper context.

Also, no need to reinvent the wheel. You’re not a psychologist, and you don’t need to be. Neurodivergent students benefit from licensed coaching and evidence-based supports. At UT, Longhorn TIES provides neurodiversity training for faculty and staff, as well as direct student support. Take the training and point students there: https://longhornties.utexas.edu/

Some minor changes in mentoring neurodivergent students can go a long way. Instead of “I simplified the tasks and they still failed,” try “Let’s co-design conditions that make time visible, tasks present, and outcomes meaningful.” That framing maintains dignity and enhances performance.

And yet, there is one important concept that I was never taught, and that I only learned in my fifties. This concept is related to how bicycles work, so let’s talk about this briefly.3 The common explanation for bicycles is conservation of angular momentum: The wheels rotate, and the angular momentum associated with this rotation conveys stability to the wheels and in turn to the bicycle as a whole.

If you try to topple a rotating wheel, the wheel instead experiences a torque at a 90-degree angle to the direction of the toppling force. Therefore, instead of falling over, bicycles will turn into the direction of the fall. And so, balancing a bicycle becomes a simple task of keeping the bike pointed into a forward direction. As long as you keep the handle bars straight the bike won’t topple. I don’t want to go into more detail of how exactly all of this works, but suffice to say the root cause is conservation of angular momentum. If you’d like to know more, read up on the concept of precession.4 You can see a demonstration of this effect in this short video:

There’s more to bicycle stability than angular momentum. If you search the scientific literature, you find all sorts of research articles on this topic.5 But here, I’m interested in a deeper, more fundamental question: Why is angular momentum conserved? In my entire high school and college physics education, I was never provided with a reason. It was just a given. Angular momentum is conserved. Just like energy, linear momentum, mass, electric charge, etc. All sorts of things are conserved, and nobody can tell us why.

Well that’s not quite true. First of all, not all of these quantities are truly conserved. And second, physicists have known about the origin of these conservation laws for nearly a century. But for some reason, it was absent from my education. I never learned about it. That is, until a couple of months ago, when I watched a video by the YouTube science channel Veritasium.

The video is titled “The Biggest Misconception in Physics,” and the thumbnail states in large font that “Energy is NOT conserved.” Both the title and the hook are a bit click-baity, even if technically true. And yet, the video is very much worth watching. We learn that conservation laws are due to symmetries. Conservation of angular momentum follows directly from the assumption (or should I say observation?) that the laws of physics are the same in every direction. Similarly, conservation of energy follows from time invariance. If the laws of physics are unchanged as time passes, then energy has to be conserved. Curiously, these invariances do not necessarily hold under general relativity, which therefore implies that energy, momentum, etc. don’t have to be conserved at sufficiently large spatial scales and/or sufficiently long times.

We understand today that symmetries are related to conservation laws, but this was not known when Einstein was developing his theory of general relativity. He knew that in his theory energy was not conserved, and he considered this to be a problem. He thought he had to find a modification of the equations so that energy would be conserved. But he didn’t know how.

The person who sorted all this out was female mathematician Emmy Noether. She realized that conservation laws were due to symmetries, and therefore if the symmetries were broken conservation laws didn’t have to apply. Energy is conserved in an empty universe, but the moment you put stuff into the universe, and the stuff starts deforming space-time, symmetry is broken and energy is no longer fully conserved. The problem was not with Einstein’s equations, but instead with the expectation that energy should be conserved.

Noether’s influence on modern physics is difficult to overstate. Modern physics, in particular quantum field theory, is all about symmetries. The symmetries in the field equations determine what elementary particles are possible and how they interact. And yet, Noether is not that well known. I had heard of the name Emmy Noether before I watched the Veritasium video, but if you had asked me what her contribution was I wouldn’t have been able to tell you. And similarly, I knew about the importance of symmetries in field theories, but I didn’t know that similar concepts lead to basic conservation laws of energy and momentum, and that Noether had first pointed out this connection.

Of course, all of this reflects my own ignorance. Many theoretical physicists are fully aware of Noether’s contributions to mathematical physics and her explanation of the relationship between symmetries and conservation laws. But I think I should have learned this in undergraduate, and I didn’t. If somebody had ever said “energy is conserved because time is invariant” I am certain I would remember. After some reflection, I have a sense of what may have caused this gap in my education. Noether’s contributions are considered advanced topics in mathematical physics, and so they are often covered only in specialized graduate classes. And even when they are covered, they are frequently described in abstract mathematical terms that obscure their importance for basic physical concepts. I may even have encountered Noether’s theorem at some point in my education and not realized its importance, and thus not remembered it. As a case in point, read the Wikipedia page on Noether’s second theorem and tell me whether the page conveys how this theorem has shaped our understanding of modern physics. To be fair, the page on Noether’s first theorem is a bit better.

Since watching the Veritasium video, I have discovered a few other videos where Noether’s work has been mentioned. This is the typical pattern where once you’re aware of something you see it everywhere. As one example, here is a video by Angela Collier where she mentions Noether’s work. (Go to 1:04:17 in the video.) Clearly Angela Collier’s physics education was better than mine.

This post was inspired by a conversation on Substack Notes, though that conversation is only tangentially related to this post. I’d like to acknowledge Billt for pointing me to the research conducted by Kooijman et al. on what makes bicycles stable.

My class is about hands-on working with and visualizing data. It inherently requires that students work on assignments at a scale of multi-day projects. Students need to familiarize themselves with a dataset, perform some exploratory investigations, create visualizations, and then document what they have done and what they can conclude from their analysis. This type of work can only be carried out in the format of a take-home assignment. And it is also something that ChatGPT or Claude Sonnet could crank out in minutes in response to a simple prompt.1

In reaction to the rise of generative AI, some teachers have transitioned their assessments to either in-class exams written by hand or oral assessments conducted in person. Neither approach works for me. The main learning outcome for my class is the ability to produce a compelling report that combines written text, computer code, and visualizations of the dataset analyzed. This skill cannot be tested with a hand-written, hour-long exam. An oral exam could make sense—I could ask students to walk me through their analysis and explain step by step what they did and why—but oral exams don’t scale. I have a hundred students in my class. I cannot possibly examine every single one of them in person.

Until recently, I felt like giving up. The only way forward I saw was to ignore the issue, teach the class as I always have, and let the dice fall where they may. Ideally, students would work on their assignments with minimal reliance on generative AI, because I would ask them to do just that. But of course I would not have any means of enforcing this behavior. And I don’t believe in AI detectors. The last thing I want to do is argue with students over whether or not they have employed AI to complete a given assignment.

However, last week I had an idea that hopefully will make the AI situation a bit more bearable. At a minimum, it will require the students to reflect on their AI use. Starting next spring,2 I will ask the students to declare, at the end of each assignment, how they used AI models in the preparation of their work. This idea is inspired by similar policies now put in place by machine learning conferences, see for example here for NeurIPS. I can’t prevent students from turning off their brain and letting the AI do all the “thinking,” but I can make them document what they did. This should prompt them to reflect on how they’re engaging with AI. And because in my class students peer-grade each others’ assignments, I hope it may make them feel uncomfortable having to admit to their peers that their entire assignment was written by AI. Maybe this will convince them to use AI a little less.

Of course all of this is based on the honor system. A student who intends to lie and deliberately obscure that their assignment was AI generated has that option. There’s not much I can do about that. But then, even without AI, students can ask their best friend or brother or hired help to write their essays and pretend it was their own work. Students who want to cheat will always find a way. There’s little I can do to prevent every such possibility. I see my job as teaching to the students that actually want to learn. My primary goal is to create a learning environment where students are motivated and inspired to make an honest effort. I hope this new AI policy will help me achieve this goal.

To get the most important issue out of the way: Executive function does not mean ability to perform like an executive. Instead, it’s the ability, broadly speaking, to organize everyday tasks and to function in the world; the capacity to get done what you need to get done when you need to do so. I could write this out in detail, but instead I’ll quote you this AI overview written by Google Gemini, which is very much on point:1

Executive function refers to the mental skills and cognitive processes that allow people to plan, organize, manage time, control impulses, and achieve goals, essentially acting as the “management system of the brain.” Key components include working memory, cognitive flexibility, and inhibition. These skills are crucial for daily tasks, work, and social interactions, and are located in the brain's frontal lobe.

What Executive Function Does

• Manages everyday tasks: It helps you plan and execute plans to get things done.

• Problem-solving: It guides your ability to figure out and solve problems.

• Attention: It allows you to focus and shift attention when needed.

• Emotional control: It helps in managing and regulating emotions.

Key Components of Executive Function

• Working Memory: The ability to hold and manipulate information in your mind.

• Cognitive Flexibility: The ability to adapt your thinking and behavior to new situations and demands.

• Inhibition: The ability to control impulses and stop inappropriate responses.

Why It Matters

• Daily living: Good executive function helps you manage your daily life, including organizing, prioritizing, and staying on task.

• Success in school and work: These skills are vital for learning, working independently, and completing large projects.

• Mental well-being: Strong executive functioning contributes to a better quality of life and can help prevent issues like anxiety and depression.

When Executive Function is Challenged

• ADHD: Attention-Deficit/Hyperactivity Disorder (ADHD) is often considered a disorder of executive functioning.

• Daily struggles: People with impaired executive function may have difficulty focusing, following directions, managing time, and controlling impulses.

• Supports: Strategies like using lists, creating routines, and organizing the environment can help individuals manage these challenges.

If you don’t have any issues with executive function, you may have read this and come to the conclusion that nearly any functioning adult must have good executive function. However, this is not the case. It is surprisingly common to see adults with some form of executive-function deficit. In particular, people on the Autism spectrum, who may be very intelligent and capable otherwise, frequently struggle with executive function.

In my work educating PhD students, I have worked with several students who struggled with executive function. This experience has led me to identify certain behavior patterns that are common among such students. One is not preparing sufficiently or appropriately for meetings or presentations. For instance, when such students give lab meeting presentations, their slides often look hastily put together, some parts are missing, slide quality varies widely throughout the slide deck, and there’s the inevitable sudden ending where the student ran out of time to prepare the final part of the talk. Another is not completing simple tasks I ask them to do. For instance, I may ask a student to read a specific paper and the next time I meet with them they haven’t read the paper but may have done all sorts of other things instead. I then ask them to please read the paper for our next meeting, and when that meeting comes around they still haven’t read the paper. And it goes on like this for weeks until eventually neither the student nor I remember why it would have made sense to read the paper in the first place.

There’s a curious observation I’ve made working with people who struggle with executive function. My natural instinct is to requests increasingly simpler tasks, but it often doesn’t help. For example, I may ask a student to do a literature review on a certain topic, and the next time we meet they haven’t done it. So then I ask them to start by reading five specific papers. The next time we meet they haven’t done it. Then I ask them to read just one paper and write a one-paragraph summary on it. The next time we meet they still haven’t done any of it. Every week, instead, they have done something else entirely. And the simpler and smaller I make the requested task, the less inclined the student seems to be to actually complete it. It’s a difficult position to be in as an advisor, as the student doesn’t do well performing large, complex tasks, but also doesn’t respond well to me trying to simplify things and reduce scope.

I have a vague notion of what may be happening in these situations: Students look at the entirety of the tasks they need to work on and discount all the items that are easy, that they know they could do relatively quickly if they actually worked on them. In their minds, because those tasks are easy, they are essentially done already. So, instead of doing those tasks, the students focus on the most difficult possible thing they could be working on, and try to make progress on that one.2 Of course, the most difficult possible thing to work on is usually too difficult for the students to actually accomplish, while the easier tasks don’t get done because they aren’t given sufficient attention or started at all. And the simpler I make the tasks I ask students to do, the lower my requests rank in the students’ internal priority ranking of things to tackle.

If you’re a student and reading this rings true to you, I’d like to ask you to consider that even simple tasks need doing and require a finite amount of time for completion. So please be honest with yourself about how long simple tasks will take and don’t wait until the last minute as you will run out of time. If you’re an advisor working with students who have some executive-function deficit, I’m not sure what to recommend. I have not found the ideal approach to managing such students. I still think breaking larger projects into increasingly simpler tasks has to be part of the solution, but maybe it needs to be accompanied by honest discussions about how long individual tasks should take, how to schedule tasks when we can roughly estimate how much time they will require, what the specific reasons are that keep students from tackling certain tasks, and so on.

To end on a positive note, I’d like to emphasize that it’s absolutely possible to be successful in life even with impaired executive function. At least four of my past PhD students have struggled with executive function and all but one of them managed to complete their PhD. Today they are successful in careers appropriate for their training, be it academia or industry. To any of my readers who may have difficulty with executive function, know that recognizing your limitations is the first step towards working around them. You can develop strategies and routines that help you get things done on time and in an organized manner, even if it doesn’t come naturally. We all have various strengths and weaknesses, and we need to learn how to work around our weaknesses while taking advantage of our own unique strengths.