This feels like a critique of stats literacy...maybe? It seems like each field develops their own favorite ways to represent common experimental data sets, but not everyone conforms and so it's on both the author to clearly explain what they're plotting and the reader to...read it. And journals to be clear about what they want to see. People get really hung up on the stuff. I'm from a math background and when working in industry I always found the plotting conventions a little silly...on the other hand I can see the utility of consistent data representations for comparing results across studies and whatnot. Idk. I don't think I ever found that these types of decisions made any difference to the takeaway of some study (e.g. in something like which method was used to produce a confidence interval or whatever).
A solution for me is to show the actual repeated fits and then their average. But, again, if one interprets things correctly, then things are useful and not misleading. So my feeling is that the issue is not reporting the variability (which is useful and good imho), but that people don't get the right meaning of what's reported.
I hate the little stars and multiple connecting bars between box and whisker plots added to show p-values. Usually with lots of comparisons between different things. Eff off. Just plot your data clearly.Choose what the actual valid comparison is.
I’d be interested to hear why you think standard error is the right choice!
I was taught to use 95%CIs in my Data Skills module but after that have always been told to use SE. I’ve asked my lecturer why and they said ‘I would always present standard error when you’re just comparing means’ which didn’t really satisfy wanting to know why.
95% CIs just look visually too large. You often end up with plots that look like the data quality is terrible even when that’s not the case. For example, two means can be significantly different even though the 95% CIs overlap.
As it happens, right now I am working on a paper that involves producing a curve from empirical data. There is naturally noise in that data. Each point in the curve is the mean of multiple data points at the same X-axis value. The standard deviation of those points looks more informative than the standard error of the mean, but it probably isn't. SEM looks so small that I expect many readers to think something is wrong.
A paper I published a few years ago is about a novel method of non-linear modeling. It shows results from synthetic data where we know the ground truth. We can generate all confidence intervals and more by using parametric bootstrapping of the synthetic data and fitting the model to each instance of the data, which varies slightly due to binomial sampling. I think that's about the best one can do in that case. We cannot know the ground truth of the real world data the method is meant to analyze so. I'd love to hear your thoughts on it.
I think the most important point is to explain carefully what you’re doing. And if there’s an aspect to the data that could cause confusion or could have the reader wonder whether you did things right it can be helpful to draw the reader’s attention to this: “Standard errors are smaller than the dots representing the point estimates. Because we used a sample size of 100, and the measurements are fairly repeatable, the uncertainty in the estimate of the mean is low.” (Or similar.)
I think an honourable mention is placing capital "T"s on top of a barplot. A bold statement against statistical pedantry and, well, common sense.
https://retractionwatch.com/2022/12/05/a-paper-used-capital-ts-instead-of-error-bars-but-wait-theres-more/
This is hilarious. I hadn’t seen it previously.
This feels like a critique of stats literacy...maybe? It seems like each field develops their own favorite ways to represent common experimental data sets, but not everyone conforms and so it's on both the author to clearly explain what they're plotting and the reader to...read it. And journals to be clear about what they want to see. People get really hung up on the stuff. I'm from a math background and when working in industry I always found the plotting conventions a little silly...on the other hand I can see the utility of consistent data representations for comparing results across studies and whatnot. Idk. I don't think I ever found that these types of decisions made any difference to the takeaway of some study (e.g. in something like which method was used to produce a confidence interval or whatever).
A solution for me is to show the actual repeated fits and then their average. But, again, if one interprets things correctly, then things are useful and not misleading. So my feeling is that the issue is not reporting the variability (which is useful and good imho), but that people don't get the right meaning of what's reported.
I hate the little stars and multiple connecting bars between box and whisker plots added to show p-values. Usually with lots of comparisons between different things. Eff off. Just plot your data clearly.Choose what the actual valid comparison is.
I’d be interested to hear why you think standard error is the right choice!
I was taught to use 95%CIs in my Data Skills module but after that have always been told to use SE. I’ve asked my lecturer why and they said ‘I would always present standard error when you’re just comparing means’ which didn’t really satisfy wanting to know why.
95% CIs just look visually too large. You often end up with plots that look like the data quality is terrible even when that’s not the case. For example, two means can be significantly different even though the 95% CIs overlap.
This is really good!
Thank you, Claus. Many excellent points.
As it happens, right now I am working on a paper that involves producing a curve from empirical data. There is naturally noise in that data. Each point in the curve is the mean of multiple data points at the same X-axis value. The standard deviation of those points looks more informative than the standard error of the mean, but it probably isn't. SEM looks so small that I expect many readers to think something is wrong.
A paper I published a few years ago is about a novel method of non-linear modeling. It shows results from synthetic data where we know the ground truth. We can generate all confidence intervals and more by using parametric bootstrapping of the synthetic data and fitting the model to each instance of the data, which varies slightly due to binomial sampling. I think that's about the best one can do in that case. We cannot know the ground truth of the real world data the method is meant to analyze so. I'd love to hear your thoughts on it.
I think the most important point is to explain carefully what you’re doing. And if there’s an aspect to the data that could cause confusion or could have the reader wonder whether you did things right it can be helpful to draw the reader’s attention to this: “Standard errors are smaller than the dots representing the point estimates. Because we used a sample size of 100, and the measurements are fairly repeatable, the uncertainty in the estimate of the mean is low.” (Or similar.)