Examples
Worked examples
- Is an instance
A reproducibility reviewer at the American Journal of Political Science who re-executed all analysis scripts before publication and certified the result
Counter-examples
Looks similar, but isn't
- Not an instance
A regular peer reviewer who happens to comment on the code is not a reproducibility reviewer in the formal role sense
Editorial commentary
A reproducibility reviewer is a reviewer whose specific role is to verify that the computational, analytic, or experimental procedures reported in a manuscript can be re-executed by another party using the provided code, data, and instructions — distinct from a content peer reviewer, who assesses the scientific merit, novelty, and interpretation of the work.
How the role differs from ordinary peer review
A content peer reviewer typically does not re-run anything; they assess whether the reported methods and results, taken at face value, support the paper’s conclusions. A reproducibility reviewer’s job is narrower and more mechanical: obtain the deposited code and data, attempt to actually execute the pipeline, and report whether it runs and whether it recreates the published numbers — the same distinction covered under computational reproducibility and results reproducibility. The reproducibility reviewer’s finding (“this ran cleanly and reproduced Table 2” or “this failed at step 3 due to a missing dependency”) is a separate, factual verification layered on top of, not a substitute for, scientific peer review.
Where this role exists in practice
This is an emerging, still-uncommon role at a specific set of journals that operate formal reproducibility checks as part of their review process — examples include the American Journal of Political Science (AJPS), Biostatistics, and the Journal of the American Statistical Association (JASA). Findings are typically published as a verification statement or badge attached to the article, and reviewers doing this work are increasingly recognised through ORCID review credit, either individually or as a journal-employed reproducibility team, rather than through the standard anonymous peer-review acknowledgement.
Sources
Center for Open Science, TOP Guidelines (Nosek et al., Science, 2015); Stodden et al., “An empirical analysis of journal policy effectiveness for computational reproducibility,” PNAS (2018).
Also known as
Code reviewer (publication) · Computational reproducibility reviewer
Machine-readable encodings
Use in your systems
<role vocab="credit"
vocab-identifier="https://casrai.org/dictionary/"
vocab-term="Reproducibility reviewer"
vocab-term-identifier="https://casrai.org/dictionary/term/reproducibility-reviewer" />{
"@context": "https://schema.org",
"@type": "DefinedTerm",
"@id": "https://casrai.org/dictionary/term/reproducibility-reviewer",
"name": "Reproducibility reviewer",
"identifier": "https://casrai.org/dictionary/term/reproducibility-reviewer",
"description": "A reviewer whose specific role is to verify that the computational, analytic, or experimental procedures reported in a manuscript can be re-executed by another party using the provided code, data, and instructions, distinct from a content peer reviewer.",
"inDefinedTermSet": "https://casrai.org/dictionary/domain/credit-extensions#set",
"url": "https://casrai.org/dictionary/term/reproducibility-reviewer",
"alternateName": [
"Code reviewer (publication)",
"Computational reproducibility reviewer"
],
"license": "https://creativecommons.org/licenses/by/4.0/",
"publisher": {
"@id": "https://casrai.org/#organization"
},
"author": {
"@id": "https://casrai.org/#editorial-team"
},
"datePublished": "2026-05-21T01:59:50",
"dateModified": "2026-09-05T14:22:01",
"inLanguage": "en-GB",
"isAccessibleForFree": true
}
