Skip to main content

Name
Michael Beyak

Wed, 12/06/2017 - 00:23

Great blog. There are a few things that I'd like to add. Along the lines of the statistical issues you've nicely outlined, there another issue that may importantly affect reproducibility and ultimately translatability. What I'm referring to is, our convention of using a "p<0.05" to indicate a result is something that is an important and "likely true." While it is true that p<0.05 means there is less than a 5% chance of finding this result if the groups are the same (i.e. null hypothesis is true), it does not mean that there is a 5% chance of "being wrong". In fact this "false discovery rate" (i.e. the chance of declaring that there is a real difference, when in fact the groups are the same, no difference) is much higher (many cases at least 30% or more especially if n values are small). It depends on the statistical power of your study and number of observations (assuming all other parts of your design are perfect). This is an excellent article by David Colquhoun (a renowned ion channel researcher) that requires a bit of reading, but outlines the issue nicely (http://rsos.royalsocietypublishing.org/content/1/3/140216). However in the scientific world, these are the expectations of journals, and reviewers. (another good article on p values and the “value of a p valueless paper” https://www.nature.com/articles/ajg2004321). Really, results should be expressed with confidence intervals that the reader can use to assess if important clinical or scientific differences exist or not. Maybe there will be a day when we publish the predicted false discovery rate based on our experiments and results. Colquhoun argues that p<0.05 means not much more than “worth another look” (bringing me to the points below). I'd also expand upon the issue of publication bias in preclinical research. As you've said above, it is much more likely to have your results published if the results are "positive". This is a well - known fact in the clinical medical literature, but I'd argue that the problem is even more pervasive in the preclinical / basic science realm. In fact not only do negative results go unpublished, in fact if the preliminary experiments turn up an initial negative result, then these lines of inquiry are often abandoned. So if only "positive results" are published, and the false discovery rates are high (for the reasons outlined above) no wonder we have a problem. This is compounded not only by the preference of positive over negative results, but also in the way we are rewarded with papers and grants when our hypothesis is "right". i.e. we got the positive results that we predicted (how often has a disappointed student, or PI said "the experiment didn't work"). Finally WRT reproducibility, experiments that confirm or refute the results of a previous study on the same topic are often not published. Such studies are deemed "not novel", or “incremental” or even "wrong" based on the fact that a previous study was published that said the opposite. One of the things we learn in science is that an idea is much more likely to be true if "independently confirmed", however there are few venues to disseminate these confirmatory results to reassure us about the "truth" of an idea, or to raise question about it if contrary results are observed. Ultimately I think over the next few decades there will be a shift away from the journal model of disseminating results, and peer review based more on the quality of the experimental design (free of bias, appropriate model) and meaningful statistical analysis. Of course then universities and granting agencies will have to look at new metrics of research contributions and productivity. Sorry for the long reply, and I'm not meaning to be too cynical about the state of preclinical, basic biomedical research (all of the pitfalls above also apply to clinical studies). We've learned a lot, and most of our major breakthroughs in medicine started in the lab. However we need to start to be cognizant of these issues and move our collective fields forward, to reach more robust and meaningful experimental conclusions.

Name
Michael Beyak
The content of this field is kept private and will not be shown publicly.

Plain text

  • No HTML tags allowed.
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.