Replicability

Replicability and reproducibility name a real and useful distinction in research methodology, but which term maps to which side of that distinction is genuinely contested, and it has flipped, by explicit committee decision, at least once in a major standards body. This entry states CASRAI’s working convention and explains the disagreement rather than presenting one side as simply correct — see the fuller reproducibility vs. replicability comparison for the side-by-side.

The National Academies of Sciences, Engineering, and Medicine’s 2019 consensus report, Reproducibility and Replicability in Science, defines reproducibility as obtaining consistent results using the same input data, computational steps, methods, and code — the computational-audit sense also used by CASRAI’s reproducibility audit entry — and replicability as obtaining consistent results across independent studies aimed at the same scientific question, each collecting its own new data. Under that convention, which this entry follows, replicability is the property tested by an independent replication study: does the effect hold up in a new sample, not merely in a re-run of the original numbers.

The Association for Computing Machinery’s own terminology ran in the opposite direction for over a decade before it changed: ACM’s earlier vocabulary defined ‘reproducibility’ as a different team obtaining the same result with a different setup, and ‘replicability’ as a different team obtaining the same result with the same setup — the reverse pairing from NASEM’s. On 24 August 2020, ACM formally swapped the two definitions to align with NASEM and with the more common usage elsewhere in science, citing exactly this kind of cross-field confusion as the reason for the change. Papers, artifact-review badges, and checklists published under ACM’s older terminology use the words in the opposite sense from material published after the 2020 change, and a reader working across computer science and other fields needs to check which convention a given source is using rather than assume.

Because of this history, CASRAI’s practice-facing guidance treats the underlying distinction, not the label, as the load-bearing fact: what matters administratively is whether a claim of “we could reproduce it” means “the deposited code and data reran cleanly” or “an independent, new sample supports the same conclusion.” These are different levels of evidentiary strength, and a research office, funder, or journal policy should specify which one it actually requires rather than relying on either single word to carry that meaning unambiguously.