Honestly this aspect is why it can be good to see a 'standard' random seed (e.g., 0, 1, 42) as it seems less likely to be cherry-picked than 17582956172.
Please let me know if you ever write a programming for biologists textbook; I learned more from this blog post than I have in multiple days of vibe coding.
I will probably never write a programming book. The problem with programming books is they are outdated the moment they go into print, as the field moves so fast. I deliberately didn't put any code into my dataviz book for that exact reason.
I'm pretty sure it was mentioned in the post. ("When I wrote my book on data visualization I used this technique quite frequently, for example in this chapter.")
Ah! I run galaxy-scale simulations in which locations of supernova are generated through a random seed. Since I'd like my simulations and therefore results to be reproducible I decided to fix a random seed. 42 since I'm a big hitchhiker's fan, without know until now, that there are many others like me.
"Using a fixed random seed when splitting data into training and test sets is uniquely bad, as you’re always going to be sampling the same split when you’re re-training your classifier."
There are certainly use cases where changing your seed is beneficial, but the main one -- comparing runs on different experiments -- requires not completely randomizing where your train/test come from between runs!
Of course, if you want some more robust metrics / information, you can always choose different train/test splits and average your results...or even better, use cross-validation. But for 'comparing across single runs'....you should never change the seed # between runs...
I don’t think you intended to imply this, but the popularity of 42 is not at all Douglas Adams’ fault; he just happened to be very good at writing the kind of joke that is outrageously funny to the kinds of people who need to set random seeds in their daily lives
“The Hitchhiker’s Guide to the Galaxy, a humorous, quirky science fiction novel. The book was very popular in the 1980s and 1990s, was adapted several times for radio and TV” — not quite. It was a BBC radio serial first - written mainly by Adams, although producer John Lloyd also has writing credits for some episodes - before Adams wrote the novels. I think the radio serials only covered the ground of the first two books.
While you hinted at this in your article, I feel compelled to note explicitly that there are times when you really should use the same seed over and over. Notably, while you are debugging code. I find it invaluable to be able to step through execution to find out why the code is failing as it does and being able to compare runs is often critical for this. Then once the you have the obvious bugs out, it is time to start changing the seed. At this point it is good to test to see that you have a seed-independent result. If you don't then that's a whole other set of problems to worry about. There are simple ways to extend the period of a PRNG that one should always use to avoid (as best you can) exacerbating correlations in your output. It is often the case that a single event will require 10 or 20 (or maybe many more) random numbers. If the period of the PRNG is too short, you end up sample planes in this 10 or 20 dimensional space rather than an unbiased, "random" selection of places. Wiping out the correlations in your generator by extending the period of repetition is, I think, as important, possibly more important, than changing seeds every run. A simple way I use is to start a run by generating a list of a few hundred random numbers and then using a random number to select which to return and replace each time. I'm sure I've told you nothing new. Just felt like getting this out there explicitly.
Come on, it is well known that you have to change your random seed until your results become significant, so obviously it can’t always be 42. /s
Ah, I didn't consider that. Good point!
Honestly this aspect is why it can be good to see a 'standard' random seed (e.g., 0, 1, 42) as it seems less likely to be cherry-picked than 17582956172.
Please let me know if you ever write a programming for biologists textbook; I learned more from this blog post than I have in multiple days of vibe coding.
I will probably never write a programming book. The problem with programming books is they are outdated the moment they go into print, as the field moves so fast. I deliberately didn't put any code into my dataviz book for that exact reason.
A very good point on the programming book, however I’m immediately hearing that there is a dataviz book
I'm pretty sure it was mentioned in the post. ("When I wrote my book on data visualization I used this technique quite frequently, for example in this chapter.")
Either way, here it is: https://clauswilke.com/dataviz/
There's also a class based on this book, and it has code examples and exercises: https://wilkelab.org/SDS366/
Ah! I run galaxy-scale simulations in which locations of supernova are generated through a random seed. Since I'd like my simulations and therefore results to be reproducible I decided to fix a random seed. 42 since I'm a big hitchhiker's fan, without know until now, that there are many others like me.
This is excellent mathematics.
Thank you.
"Using a fixed random seed when splitting data into training and test sets is uniquely bad, as you’re always going to be sampling the same split when you’re re-training your classifier."
There are certainly use cases where changing your seed is beneficial, but the main one -- comparing runs on different experiments -- requires not completely randomizing where your train/test come from between runs!
Of course, if you want some more robust metrics / information, you can always choose different train/test splits and average your results...or even better, use cross-validation. But for 'comparing across single runs'....you should never change the seed # between runs...
I don’t think you intended to imply this, but the popularity of 42 is not at all Douglas Adams’ fault; he just happened to be very good at writing the kind of joke that is outrageously funny to the kinds of people who need to set random seeds in their daily lives
“The Hitchhiker’s Guide to the Galaxy, a humorous, quirky science fiction novel. The book was very popular in the 1980s and 1990s, was adapted several times for radio and TV” — not quite. It was a BBC radio serial first - written mainly by Adams, although producer John Lloyd also has writing credits for some episodes - before Adams wrote the novels. I think the radio serials only covered the ground of the first two books.
While you hinted at this in your article, I feel compelled to note explicitly that there are times when you really should use the same seed over and over. Notably, while you are debugging code. I find it invaluable to be able to step through execution to find out why the code is failing as it does and being able to compare runs is often critical for this. Then once the you have the obvious bugs out, it is time to start changing the seed. At this point it is good to test to see that you have a seed-independent result. If you don't then that's a whole other set of problems to worry about. There are simple ways to extend the period of a PRNG that one should always use to avoid (as best you can) exacerbating correlations in your output. It is often the case that a single event will require 10 or 20 (or maybe many more) random numbers. If the period of the PRNG is too short, you end up sample planes in this 10 or 20 dimensional space rather than an unbiased, "random" selection of places. Wiping out the correlations in your generator by extending the period of repetition is, I think, as important, possibly more important, than changing seeds every run. A simple way I use is to start a run by generating a list of a few hundred random numbers and then using a random number to select which to return and replace each time. I'm sure I've told you nothing new. Just felt like getting this out there explicitly.
I feel personally attacked.