With regard to sample size, small sample sizes in studies that are taken at face value is an issue. Your example of small sample sizes in the context of less-serious exploratory research would not be the kind of thing that would get press, unless someone then treated that exploratory research as canon.

My understanding of the replication crisis, however, comes from people like Goldacre & Neuroskeptic who write frequently about the subject; their coverage focuses on systematic distortions in scientific literature related to perverse incentives. Your post here is the first one I’ve seen that situates it as a problem related to specific studies misleading primarily other scientists in the field. I’d be interested in knowing what articles in particular you’re complaining about, because the suggestion that any single exploratory study in isolation is a problem (outside of the tendency for the science press to hype low-quality studies) tends to be dismissed early in the coverage I’ve read. (In other words, I’ve only seen this framed as a systematic incentive problem, where scientists are discouraged from performing certain forms of vital tasks, such as large-scale replications with large samples.)

Thanks to your response, I’m more willing to treat the article in good faith, but my alarm bells started going off specifically because the model you present of the reasons behind the replication crisis is one that is explicitly rejected in the pro-replication-movement pieces I thought were seminal.

Written by

Resident hypertext crank. Author of Big and Small Computing: Trajectories for the Future of Software. http://www.lord-enki.net

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store