Irreproducible Research: The Need for Study Validation

MODERN SCIENCE is facing what could be the gravest threat to its remarkable run of success — a run that extends from the beginning of the Scientific Revolution some four and a half centuries ago. The issue is the “reproducibility crisis” in which a significant percentage of scientific studies are unable to be reproduced by others, not only undermining their value but also threatening the public’s faith in scientific research as a whole.

Many explanations are being offered as both the popular and academic press take an increasingly skeptical look at the issue. The size of the problem is not yet clearly known, nor even agreed upon. Nor are the root causes or potential solutions matters even close to consensus. But what nobody seems to be disputing is the reality that a huge swath of scientific research — including that being conducted in many medical fields — is basically useless. More ominously, it is also not widely disputed that the reproducibility crisis threatens the very foundations of the scientific culture in the West, upon which rests modern medicine.

Many science and medical writers are warning that the reproducibility crisis undercuts the very premise of the scientific method: the idea that science is ultimately based on facts that can be proven through observation and/or experimentation. If much of that experimentation is so flawed as to prove nothing, or if the analysis of the observations is statistically meaningless, then our science is far less efficient and effective than we have all believed.

How It’s Supposed to Work

Everyone working in the medical sciences is well aware of how the scientific method works: Let the evidence guide you where it will. Either a drug works against a disease, or it does not. A treatment leads to improvement, or it does not.

A properly designed study will account for all outside influences, and it will compare a control group vs. a study group. Further, in most reputable journals, no studies can be published until they’ve been reviewed by other experts in the same discipline. This “peer review” process is designed to be the main defense against fraud and incompetence. These peers should be reviewing a study to ensure it was properly configured to account for all variables so that the experiments truly indicate whether the drug or treatment being tested is effective. They should also be weighing whether any conclusions tied to the study are truly reflective of the evidence offered.

But it is this entire process that is under heightened scrutiny as hundreds, and potentially thousands or more, of research papers are being found wanting in one respect or another.

The Scientific Challenges

In the last few years, researchers from a variety of disciplines discovered that while reviewing previous work in their fields, they were unable to get the same results as reported in published papers. Either primary research or statistical conclusions drawn from existing research were found to be impossible to reproduce. Some researchers have been worried about this phenomenon for a while, even if the issue is only now gaining traction in the academic press.

The advocacy nonprofit Public Library of Science (PLOS) published an article 11 years ago arguing that statistical claims associated with research results were often invalid or misleading. Its author, John P. A. Ioannidis, a statistician and physician on faculty at Stanford, argues that most often it is not fraud that leads to bad studies, but the research culture itself. His belief is that out-and-out fraud is but a small sliver of bad research; instead, secrecy and misapplication of statistics theory are the main culprits.¹ (Outright fraud is easier to detect and confront than systemic bias. There is even an entire website devoted to reporting on retraction of studies at retractionwatch.com.) Roger Peng, associate professor in the department of biostatistics at Johns Hopkins Bloomberg School of Public Health, backs up Ioannidis’ assertion that statistics illiteracy clouds far too many conclusions drawn from research.²

Statistical error may come from not understanding how to properly account for anomalies that arise during the course of an experiment. For instance, PLOS is conducting an analysis of how researchers account (or fail to account) for rodents that die from seemingly unrelated issues during the course of medical experiments.³ An earlier review of experiments that utilized rodents found a significant number had a smaller number of test subjects at the conclusion of the research than at the beginning, with no accounting for this discrepancy.

Beyond errors that escape peer review are the pressures associated with the highly competitive, cutthroat world of academic research in the United States and Western Europe. With every faculty member and tenure-track adjunct having to conduct original research in order to get or keep a job in their chosen field — or simply to finish their doctorate program — the pressure to conduct original research is enormous. As noted in 2013 by Fiona Fidler and Ascelin Gordon on the Phys.org website, “There can be little doubt that the ‘publish or perish’ research environment fuels this fire. Funding bodies and academic journals that value ‘novelty’ over replication deserve blame too.”⁴

The growing number of participants in science and academia only adds to these pressures. There are far more working-class and middle-class students qualifying for and being accepted into four-year universities today than was true a half-century ago — a 300 percent increase in the number of college and university students in the United States, far outstripping population growth.⁵ The United States alone has gone from turning out an average of 545 science and engineering doctorates a year in the 1920s to more than 27,000 in 2010.⁶

While this growth has provided an undoubted burst of democratization to what was formerly a preserve of the rich, we now also have more and more researchers fighting for grant money and tenure-track positions, increasing competitive pressures. Just in the years from 1997 to 2011, funding requests to the National Institutes of Health doubled, from 31,000 research grant applications to 62,000.⁶

The result of more people competing for the same resources is probably predictable. Chris Chambers, professor of psychology and neuroscience at Cardiff University in Britain, said “significance chasing” — aiming for the highest perceived level of interest in order to attract more attention and funding — is inadvertently leading to bad research.⁷ (Ironically, his March 2015 comments came at a conference at University College London just a week before BioMed Central, a major publisher of medical journals, announced the retraction of 43 published papers for reasons of peer review fraud.8 It was a BioMed Central blog that quoted his comments made at the conference.)

The anonymous “Neuroskeptic” blogger for Discover magazine, as well as one of his regular readers responding in the comments section of his post, wonders if the fact that most academic and scientific journals publish only “significant” results doesn’t also add to the pressure through what Neuroskeptic refers to as “publication bias.” Neuroskeptic also worries about “p-hacking,” often referred to as data dredging, in which existing data is automatically (via computer algorithms) searched for statistical anomalies. Rather than searching for evidence to back up or dispute an existing hypothesis, the patterns themselves are the subject of the search — and once found, hypotheses are then developed to explain them.⁹

All of these different pressures — to get published, to stand out from other researchers, to secure funding — are likely introducing unintended bias into the conclusions reached, if not the very research itself. And yet, just as the ability to reproduce a study would seem to be more important than ever, these same pressures faced by academics are leading many of them to show great reluctance to share details of their research. This protects their intellectual work, but also makes it near impossible for others to replicate their work. As Fidler and Gordon pointed out in their Phys.org commentary, “Data sharing and other procedures outlined here can be time-consuming and currently provide little academic reward.”⁴

The Cultural Challenge

Coverage of these issues has begun leaking over to the mainstream press. The first big waves in the media came in 2012 when pharmaceutical researchers C. Glenn Begley and Lee Ellis dove into 53 supposedly groundbreaking oncology studies from 2001 to 2011, and could only reproduce 11 percent.¹⁰ Drug companies took notice, as they were pouring billions of dollars in private research money into new studies designed to build on the results of earlier — suddenly questionable — studies.

More reports about irreproducible research followed in the popular media. Time magazine weighed in on the issue in 2014,¹¹ as did Wired.¹² Last July, the popular science blog I09 published a piece titled “Half of Biomedical Research Studies Don’t Stand Up to Scrutiny.”¹³ Smithsonian magazine addressed it earlier this year.¹⁴

These reports might not sound as ominous as the more rigorous scientific papers, but public support for the sciences is essential to preserving or even increasing government funding of research. Most funding for basic scientific research comes from governments, with grants awarded based on a combination of past success and the potential for gains in new knowledge.

It is a shared cultural belief in the ability of science to provide important advances in our understanding of the universe, and apply those new insights in ways that improve our quality of life, that makes it possible for the government to invest heavily in scientific research — particularly medical research. Threaten that belief in the legitimacy and efficacy of scientific research, and you threaten public funding. Without voter support for government funding, said funding cannot survive — particularly in democratic systems, and certainly not at the levels to which we have become accustomed.

Finding Solutions

It is likely that further study of the issue is needed before a consensus emerges on the scope and nature of the problem. And without a broader consensus, it will be difficult to change the overarching culture of scientific research (including the disinclination to full disclosure of study parameters).

Still, changes in the way research is conducted — or at least funded and published — are already underway. The National Institutes of Health (NIH) has beefed up its grant application process, requiring more explanation of the science behind the proposed study and more rigorous efforts to eliminate variables.¹⁵ This, of course, applies only to medical studies, but the NIH standards will encourage other government agencies to at least take notice.

As gatekeepers of information about studies, many medical and academic journals are changing the way they accept papers for publication. PLOS now requires full disclosure of all data before it will publish any research studies.¹⁶ And Nature magazine is offering data repository agreements to encourage public sharing of research data among its contributors.¹⁷ Nature has gone so far as to devote an entire online hub to the topic at www.nature.com/news/reproducibility-1.17552.

And the debate on what else ought to be done continues. Neuroskeptic proposes peer review of research studies before they even begin, with journals committing to publishing the results no matter what they are.¹⁸ Peng, the Johns Hopkins biostatistician, is proposing enhanced instruction in statistics for budding researchers in all scientific disciplines to improve the quality of conclusions reached from study results.¹⁹ Fidler and Gordon suggest a reproducibility index, which they argue would require more sober statistical analysis of research results. They also propose that researchers share their computer code — their search algorithms — along with the data used in the study being reported on, so that others can provide a true “apples-to-apples” comparison.⁴

It will likely take a combination of all these proposals to begin changing the culture of medical and scientific research. But with so many billions of dollars at stake in both private and public research funds, the current uproar over the “reproducibility crisis” is unlikely to lessen until system reforms are put in place.