Reproducibility

Reproducibility is the ability to obtain consistent computational or analytical results when the same data and analysis procedures are applied, typically by an independent investigator using the same code and tools. It describes a property of the study record — whether the reported numbers can be regenerated from the deposited data and code — not a property of the underlying scientific claim itself.

Reproducibility vs. replicability vs. repeatability

Terminology in this area varies by field and even by era, which is itself a recurring source of confusion in the methods literature. CASRAI follows the U.S. National Academies of Sciences, Engineering, and Medicine (NASEM) 2019 consensus report, Reproducibility and Replicability in Science, as the reference framework:

• Reproducibility — obtaining consistent results using the same input data, computational steps, methods, code, and conditions of analysis. Only the analyst changes; the data and methods stay fixed. NASEM calls this, more precisely, computational reproducibility — see CASRAI’s Computational reproducibility entry.
• Replicability — obtaining consistent results using new data, and potentially different methods, in a study aimed at the same scientific question. Here the data changes; only the scientific question stays fixed. See also Replication study.
• Repeatability — an older, narrower usage from metrology and the ACM’s artifact-review vocabulary: the same team, using the same experimental setup, obtains consistent measurements across repeated trials. NASEM’s 2019 report explicitly notes that some research communities use ‘reproducibility’ and ‘replicability’ in the opposite sense from the definitions above — always confirm how a specific journal, funder, or field defines its terms rather than assuming NASEM’s usage applies universally.

A result can be reproducible (the numbers regenerate exactly from the same inputs) without being replicable (the effect does not reappear with new data) — reproducibility is necessary but not sufficient for a finding’s credibility. See Reproducibility crisis for the wider debate this distinction sits inside, and Generalisability and Robustness for two further, still-distinct properties a finding can have.

Why reproducibility matters

Reproducibility is the minimum, verifiable check on a published result: if an independent analyst cannot regenerate the reported numbers from the deposited data, code, and stated procedure, the record cannot yet be assessed for anything further, including replicability. Three practical consequences follow. First, funders increasingly require it directly — the NIH Rigor and Reproducibility policy (in force since 2016) requires grant applications to address the scientific premise, rigorous experimental design, consideration of relevant biological variables, and authentication of key resources. Second, it protects downstream reuse: a data management plan, method, or dataset that others cannot reproduce cannot be safely built on, which wastes both grant funding and researcher time on follow-on work built over an unverified base. Third, it underpins the credibility of the published record itself — reproducibility failures documented across fields (see Reproducibility crisis) are what motivated the current wave of open-science reforms, including data availability requirements and computational-workflow standards.

How reproducibility is assessed

In practice, reproducibility is checked, not assumed. Common mechanisms include:

• Reproducibility audits — an independent party (a journal, a dedicated reproducibility reviewer, or a lab’s own QA process) re-runs the deposited analysis end to end and compares outputs to the published results.
• Reporting checklists attached to submission, such as the Nature Portfolio Reporting Summary, which force disclosure of the specific inputs (sample sizes, software versions, statistical methods) an independent reproduction would need.
• Transparency standards such as the TOP Guidelines (Transparency and Openness Promotion), which set graded disclosure requirements a journal or funder can adopt for data, code, and materials citation.
• Computational infrastructure — containerized environments, workflow managers, and continuous-integration pipelines that rebuild a paper’s results from raw data on every change. See CASRAI’s guide to reproducibility infrastructure: workflows, containers, and code sharing.
• Data availability statements that specify exactly where the data and code live and under what access conditions — see CASRAI’s guide on writing a data availability statement for reproducibility.
• Badging schemes that publicly certify a paper met a defined reproducibility bar, tracked under CASRAI’s Reproducible Research Practices (RRP) entry.

Because reproducibility is assessed at different levels of a study, CASRAI’s dictionary distinguishes several related sub-types worth cross-checking depending on what exactly is being verified: Methods reproducibility (can the procedure itself be repeated as described), Results reproducibility (do independent execution of the same procedure produce corroborating results), Inferential reproducibility (do independent analysts draw the same qualitative conclusions), and Empirical reproducibility (sufficient methodological detail is reported to enable an attempt at all) — a taxonomy originally set out by Goodman, Fanelli, and Ioannidis (2016).

References

• National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science (2019).
• Goodman, Fanelli, and Ioannidis, ‘What does research reproducibility mean?’, Science Translational Medicine 8(341), 2016.
• NIH, Rigor and Reproducibility policy.