This guide is intended for researchers needing an understanding of reproducibility to improve their own research or to meet funder requirements related to reproducibility. The intent is to assist both early career researchers as well as more experienced researchers.
Although this guide will provide information about reproducibility as well as tools and resources that can assist in meeting the conditions for reproducibility, it is not designed to include granular guidance on reproducibility. Wherever possible, resources providing more in-depth information, specific to discipline and study methods, will be linked.
Reproducibility and replicability are often discussed when describing the extent to which research findings can be confirmed by others. In many cases, the terms are used interchangeably or take on different meanings according to the discipline describing the concepts. This guide adopts (and adapts) the definitions crafted by the National Academy Sciences in their report, Reproducibility and Replicability in Science (2019):
Reproducibility is the ability to take the same data, software, code, and analysis methods created and used for a completed study and generate the same results reported in the study.
Replicability is the ability to take the same research question and research methods used in a published study and come up with results reasonably similar to those found in the original study.
Reproducibility requires two broad conditions: clear and transparent reporting within the published information for a given study and discoverable, accessible data and other products of research underlying the published record of a study.
Replicability also relies upon clear and transparent reporting of methods in the original study, but may be influenced by other factors, such as naturally occurring variability in populations or conditions being studied, complex systems, or through poor methodological practices, whether intentional or unintentional.
Since the early 2000s, various authors, publishers, and funders have raised concerns about the 'reproducibility crisis' across disciplines. Broadly, research stakeholders have been alarmed at the extent to which the findings in published research cannot be computationally reproduced and the results of large and important studies confirmed through replication.
Rates of irreproducibility vary by study and discipline. One review of rates of irreproducibility in preclinical sciences conservatively estimated that roughly 50% of preclinical studies are not reproducible. Within political science, the Quarterly Journal of Political Science requires 'replication packages' that allow the journal to verify that the results and figures in the submitted manuscript can be reproduced. The editors note that the vast majority of submitted manuscripts require correction and resubmission of replication materials, suggesting high rates of irreproducibility exist without processes in place to verify reported results and figures.
The inability to validate and verify published research poses considerable concern in three overarching areas:
While some cases of irreproducibility and non-replicability may arise from deliberate actions on the part of researchers, outright research misconduct makes up a very small percentage of the cases of irreproducibility and non-replicability. Generally, issues arise from honest mistakes as well as shortcomings in commonly accepted practices across scientific disciplines. The remainder of this guide seeks to outline common sources of irreproducibility and non-replicability and provide suggested methods and tools for addressing the issues underlying irreproducibility and non-replicability.
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015). The Economics of Reproducibility in Preclinical Research. PLOS Biology, 13(6), e1002165. https://doi.org/10.1371/journal.pbio.1002165
Jacoby, W., Lafferty-Hess, S., & Christian, T.-M. (2017, July 17). Should Journals Be Responsible for Reproducibility? | Inside Higher Ed. REthinking Research. https://www.insidehighered.com/blogs/rethinking-research/should-journals-be-responsible-reproducibility
National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25303.