Dictionary domain · Track D · Active
Reproducibility
Operational terminology for the practices, standards, and metadata that support reproducibility, replicability, and robustness of research outputs.
For implementers
Operational deployment checklist for the Reproducibility vocabulary: prerequisites, five deploy steps, integration notes for DSpace, Pure, OJS, Software Heritage, and OSF, plus the pitfalls that recur in the field.
What this domain covers
The Reproducibility domain assembles the vocabulary that lets funders, journals, institutions, and researchers talk about the same things when they invoke “reproducible research”. Its remit spans six adjacent areas that are routinely conflated in practice.
- Outcome distinctions — reproducibility, replicability, and robustness as three formally separate constructs, following the National Academies framing (Goodman, Fanelli, and Ioannidis 2016; NASEM 2019).
- Reporting standards — the TOP, ARRIVE, CONSORT, PRISMA, and STROBE families curated by the EQUATOR Network and the Center for Open Science.
- Methods transparency — preregistration, registered reports, protocol publication, and the data availability statement as the now-canonical disclosure surface.
- Outputs — datasets, code, materials, and computational environments captured as containers, Jupyter notebooks, and workflow artefacts under FAIR4RS-compatible practice.
- Assessment — replicability rate, robustness checks, multiverse analysis, and specification curve analysis as the techniques that quantify how stable a finding actually is.
- Governance — the Reproducibility Crisis discourse and its intersection with the research-integrity domain, including the boundary between “non-replication” and “misconduct”.
19 entries
Terms in this domain
Each entry has an operational definition, worked examples, related terms, and stable URIs. Pages link only where the term entry exists; the rest are populated as the working group ratifies each definition.
- Reproducibility
- Obtaining consistent results using the same input data, computational steps, methods, and conditions of analysis (NASEM 2019).
- Replicability
- Obtaining consistent results across studies aimed at answering the same question with new data or methods.
- Robustness
- Stability of a finding under reasonable variation in modelling choices, samples, or analytic specification.
- Pre-registration
- Time-stamped declaration of hypotheses, design, and analysis plan prior to data observation.
- Registered report
- A two-stage publication format in which a protocol is peer-reviewed and accepted in principle before results exist.
- Data availability statement
- Mandatory disclosure declaring where, how, and under what conditions underlying data are accessible.
- Computational reproducibility
- Bit-for-bit or equivalent re-execution of an analysis from the same code, data, and computational environment.
- Methods transparency
- Reporting of materials, procedures, and decisions in sufficient detail to permit independent scrutiny.
- Open materials
- Public release of study materials (instruments, stimuli, protocols) under a reuse licence.
- Open code
- Public release of analysis code under an OSI-approved licence, ideally archived with a persistent identifier.
- Protocol publication
- Formal, citable publication of a study protocol prior to or alongside primary results.
- Multiverse analysis
- Reporting the distribution of results across the full set of defensible analytic specifications (Steegen et al. 2016).
- Robustness check
- An auxiliary analysis demonstrating that the headline result survives a substantive change in specification.
- Specification curve analysis
- A systematic descriptive technique that visualises results across all reasonable analytic specifications.
- Replication study
- A study explicitly designed to re-test the inference of a prior study, classified as direct or conceptual.
- Reporting guideline
- A structured checklist defining the minimum reporting required for a study design (ARRIVE, CONSORT, PRISMA, STROBE, etc.).
- Authentication of key resources
- NIH Rigor & Reproducibility requirement to validate the identity and quality of biological and chemical resources used.
- Scientific rigor
- The strict application of the scientific method to ensure unbiased, well-controlled experimental design (NIH).
- Reproducibility crisis
- Discourse term for the empirically observed gap between published findings and successful replication across fields.
Term pages are populated as the Reproducibility working group reviews each definition. See /dictionary/contribute to propose a term.
Cross-domain reference: FAIR principles assessment (Data infrastructure domain).
82 live entries
All terms in this domain
The complete, live list of Reproducibility terms held in the CASRAI Dictionary, pulled from the published taxonomy. Each links to its full term page with the operational definition, related terms, and stable URIs.
CONSORT Flow Diagram for Randomised Trials
The figure required by the CONSORT statement (2025 item 22a) that accounts for every participant in a randomised trial across four stages — enrolment, allocation to intervention, follow-up, and data analysis — reporting a count in each box and, under item 22b, a stated reason for every loss or exclusion occurring after randomisation.
jamovi
jamovi is a free, open-source statistical software application that provides a spreadsheet-style graphical interface to analyses executed by R packages underneath it (the jmv R package for its core menu), distributed under the AGPL-3.0 licence (select components GPL-2.0-or-later) for Windows, macOS, and Linux, plus a browser-based cloud version, with no licence fee, seat count, or activation step required.
Newcastle-Ottawa Scale (NOS)
The Newcastle-Ottawa Scale (NOS) is a star-rating quality-assessment tool used to appraise non-randomized studies -- cohort studies and case-control studies -- for inclusion in systematic reviews and meta-analyses. Developed by Wells, Shea, O'Connell, Peterson, Welch, Losos and Tugwell through a University of Newcastle (Australia) / University of Ottawa (Canada) collaboration and maintained by the Ottawa Hospital Research Institute, it scores studies across three domains -- Selection (max 4 stars), Comparability (max 2 stars), and Outcome/Exposure (max 3 stars) -- for a maximum of 9 stars. It is commonly paired with the MOOSE reporting guideline for meta-analyses of observational studies, and is distinct from ROBINS-I (Cochrane's preferred non-randomized-study risk-of-bias tool) and GRADE (which rates the overall evidence base rather than individual studies).
Defined Daily Dose (DDD)
The Defined Daily Dose (DDD) is a WHO-assigned statistical unit defined as the assumed average maintenance dose per day for a drug used for its main indication in adults. DDDs are assigned only to drugs with an ATC (Anatomical Therapeutic Chemical) classification code, one per code and route of administration, by the WHO Collaborating Centre for Drug Statistics Methodology in Oslo. DDD is a fixed technical unit for measuring and comparing aggregate drug consumption across populations, institutions, and countries -- commonly expressed as DDD per 1,000 inhabitants per day -- and does not necessarily correspond to the Prescribed Daily Dose (PDD) an individual patient actually receives.
Observer Bias
Observer bias is a systematic distortion introduced by the person doing the observing, recording or interpreting, rather than by the participants or the sampling. The Catalogue of Bias defines it as a 'systematic difference between a true value and the value actually observed due to observer variation'. It arises wherever measurement requires judgement - reading an image, scoring a scale, coding an interview transcript, grading a lesion, deciding whether an adverse event is related - and it is strongest where the observer knows which group a participant belongs to and has an expectation about what they should find. It is a form of detection or measurement bias and belongs to the recording end of a study, not the sampling end: it is distinct from selection bias, which distorts who enters the study, and from confounding, which distorts the comparison itself. It is also distinct from the observer effect, where the participant changes behaviour because they know they are being watched; in observer bias the participant behaves normally and it is the observer's record of that behaviour that is wrong. Observer bias is controlled at the design and procedure level - by blinding outcome assessors, separating the people who know exposure status from the people who record outcomes, pre-specifying operational definitions and coding rules, using independent duplicate coding, and quantifying agreement between observers - rather than by any statistical adjustment after the fact.
Regression to the Mean
Regression to the mean is the tendency for an extreme measurement to be followed by one closer to the average, purely because the extreme value was partly the product of chance. It is a property of repeated measurement rather than a real change in the thing being measured, and it operates whenever a group is selected on the basis of an extreme score. A before-after comparison that selects on an extreme baseline and lacks a control group cannot distinguish regression to the mean from a treatment effect.
STAR Methods
STAR Methods (Structured, Transparent, Accessible Reporting) is Cell Press's standardized methods-section format for life-science papers. It is organized around a Key Resources Table that lists every reagent, model organism, cell line, software tool, and dataset used in a study alongside a source and, wherever one exists, a unique catalog number or Research Resource Identifier (RRID), so materials can be independently sourced and the reported experiments reproduced.
JOSS (Journal of Open Source Software)
JOSS is a peer-reviewed, diamond open-access academic journal, founded in 2016 under the Open Journals umbrella, that publishes short papers (roughly 750-1750 words) about research software rather than the software itself. Acceptance requires an OSI-approved open-source licence, substantial and feature-complete engineering work, adequate documentation and tests, and a version-specific DOI from an archiving service (e.g. Zenodo or figshare) obtained after review; accepted papers receive their own Crossref DOI.
Citation File Format (CITATION.cff)
A CITATION.cff file is a plain-text, YAML-formatted metadata file, conforming to the Citation File Format schema, placed in the root of a code (or dataset) repository to record machine- and human-readable citation information -- authors (ideally with ORCID iDs), title, version, release date, identifiers such as a DOI, and licence -- so tools and humans can produce a correct, consistent citation without guesswork.
Reporting Guidelines
<p><strong>Reporting guidelines</strong> are checklists, flow diagrams, or structured-text tools that specify the minimum set of information authors must include when writing up a specific type of study, so that readers, peer reviewers, and editors can judge what was done and trust that nothing material has been left out. The EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network, an international initiative founded in 2006 and based at the University of Oxford's Centre for Statistics in Medicine (NDORMS), is the field's central clearinghouse: it defines a reporting guideline as a checklist, flow diagram, or structured text developed using an explicit, documented consensus methodology to guide authors in reporting a specific type of research, and it maintains a searchable library of several hundred such guidelines spanning study designs, specialties, and report sections.</p><p>A reporting guideline is not a methodological standard. It does not tell an investigator how to design a trial, calculate a sample size, randomize participants, or run an assay -- that is the job of methodological/conduct standards such as ICH Good Clinical Practice, CIOMS guidance, or a funder's data-collection protocol. Nor is a reporting guideline a critical-appraisal or risk-of-bias tool -- it does not score how well a study was conducted (that is the role of instruments such as the Cochrane risk-of-bias tools). A reporting guideline answers a narrower, prior question: <em>given that the study was conducted a particular way, has the write-up disclosed enough about it -- the eligibility criteria, the randomization method, the funding source, the flow of participants -- for a reader to evaluate and, ideally, reproduce the work?</em> An author can follow a reporting guideline to the letter while having run a methodologically weak study, and a well-conducted study can still be reported so incompletely that a reader cannot tell.</p>
Cochrane Handbook for Systematic Reviews of Interventions
The Cochrane Handbook for Systematic Reviews of Interventions is the methodological standard maintained by Cochrane (founded 1993, named after epidemiologist Archie Cochrane) that specifies how a systematic review of a health intervention must be planned, conducted, and reported to qualify as a Cochrane Review. It requires a registered protocol with a PICO-structured question, a comprehensive pre-specified search strategy across multiple databases, dual independent screening and data extraction, formal risk-of-bias assessment (the RoB 2 tool for randomized trials, ROBINS-I for non-randomized studies), GRADE-based certainty-of-evidence rating, and, where studies are sufficiently similar, meta-analysis. Reviews that follow the Handbook are published in the Cochrane Database of Systematic Reviews (CDSR) within the Cochrane Library and are maintained as living documents, updated as new evidence emerges.
PICOTS Criteria
A research question is PICOTS-structured when it explicitly specifies all six elements — Population, Intervention, Comparator, Outcome, Timing, and Setting — rather than only the four PICO elements, typically to derive study-eligibility criteria for a systematic review, evidence synthesis, or health-technology assessment protocol.
Nature Portfolio Reporting Summary
A mandatory, standardized fillable-PDF form Nature Portfolio requires with manuscript submission for life sciences, behavioural & social sciences, or ecological/evolutionary/environmental sciences research articles, disclosing statistics reporting, software/code, data availability, and a field-specific study-design track (sample size, data exclusions, replication, randomization, blinding, or their track-specific equivalents), with negative disclosures required rather than fields left blank; the completed form is shared with editors/reviewers during assessment and published alongside the accepted paper.
ACM Artifact Review and Badging
ACM Artifact Review and Badging is a formal peer-evaluation process, run alongside conventional paper peer review at ACM-sponsored conferences and in ACM journals, in which authors of a computer-science research paper voluntarily submit their supporting 'artifacts' -- source code, datasets, containers, build scripts, or other digital objects needed to run the reported experiments -- to a separate Artifact Evaluation Committee (AEC). The AEC checks whether the artifact is documented and runnable, and in stricter tracks, whether running it actually reproduces the paper's reported results. A paper that passes earns one or more standardized badge icons (Artifacts Available, Artifacts Evaluated -- Functional, Artifacts Evaluated -- Reusable, Results Reproduced, Results Replicated), printed on the published paper itself as a visible, per-paper certification distinct from the paper's own peer-review outcome.
P-Value
The p-value is the probability of observing a result at least as extreme as the one found in a sample, calculated on the assumption that the null hypothesis is true. It is not the probability that the null hypothesis is true, and it does not measure effect size or practical importance — only how surprising the observed data would be if there truly were no effect.
Sampling Methods: Probability and Non-Probability Types Explained
A sampling method is the defined procedure a study uses to select which members of a population are observed or measured. It qualifies as a probability method only if every population member has a known, non-zero chance of selection via a defined chance mechanism (simple random, stratified, systematic, or cluster sampling) -- this is what supports formal statistical inference and generalizability claims back to the population. Non-probability methods (convenience, purposive/judgmental, snowball, quota sampling) select participants without a defined chance mechanism, which is legitimate and standard in qualitative and exploratory research but does not, on its own, support formal population-level generalization.
Confidence Interval
A confidence interval is a range of values, calculated from sample data using a defined estimation procedure (point estimate plus a margin of error derived from the sampling distribution of the estimator), constructed at a stated confidence level (commonly 95%). The confidence level describes the long-run reliability of the construction procedure across repeated sampling -- the proportion of such intervals that would contain the true, fixed population parameter if the sampling were repeated many times -- not the probability that this specific interval contains the parameter. A range lacking a stated confidence level and a defined estimation procedure (e.g. a bare sample min-max) is not a confidence interval.
Effect Size
Effect size is a standardized, quantitative measure of the magnitude of a difference, relationship, or association observed in a study, reported independently of sample size and distinct from statistical significance. Where a p-value indicates only whether an observed effect is unlikely to be due to chance, an effect size (e.g., Cohen's d, Pearson's r, an odds ratio) indicates how large or practically meaningful that effect actually is. The APA Publication Manual (7th edition, Section 6.5) requires reporting an effect size and confidence interval for each primary outcome, and most major journals across the behavioral, social, and biomedical sciences hold manuscripts to an equivalent expectation.
Operational Definition
A statement that specifies the exact, observable procedure used to measure or manipulate an abstract construct or variable within a specific study, so that another investigator could apply the same procedure to the same phenomenon and obtain comparable data. An operational definition is distinct from a conceptual (or constitutive) definition, which states what a construct means in theory without specifying how it will be observed; a definition only counts as operational if it names the instrument, coding rule, threshold, or manipulation used, not just the general idea being studied.
Blinding and Masking in Clinical Trials
Blinding (or masking, the term FDA/ICH guidance now prefer) in a clinical trial is the procedure of keeping one or more trial parties -- the participant, the investigator/care provider, the outcomes assessor, and/or the data analyst -- unaware of which treatment arm a given participant was assigned to, from the point of randomization onward. It is recorded on ClinicalTrials.gov as the "Masking" data element with levels of None (open-label), Single, Double, Triple, or Quadruple, depending on how many of those roles are kept unaware. Masking is a distinct methodological safeguard from randomization and allocation concealment: it addresses performance bias (differential behavior or care from knowing the assignment) and detection/ascertainment bias (differential outcome judgment), not selection bias at enrollment.
PRISMA-ScR (PRISMA Extension for Scoping Reviews)
A report is PRISMA-ScR compliant when it satisfies PRISMA-ScR's 20 essential reporting items (Tricco et al. 2018), frames its question in PCC (Population, Concept, Context) rather than PICO terms, and documents evidence "charting" and mapping rather than risk-of-bias-weighted synthesis -- the checklist purpose-built for scoping reviews, distinct from the core 27-item PRISMA 2020 checklist used for systematic reviews of intervention effects.
Open Science Badges (Open Data, Open Materials, Preregistered)
The Open Science Badges are a set of journal-awarded visual icons, created and administered by the Center for Open Science (COS), that certify a published article met a disclosed openness standard for one of three practices: publicly posting the study's underlying data (Open Data), publicly posting the materials needed to reproduce the reported procedure and analysis (Open Materials), or registering the study design -- and, for the Preregistered+Analysis Plan variant, the analysis plan -- in a public registry before data collection began (Preregistered). A journal awards a badge either on a simple author self-disclosure statement or after independent editorial verification, and displays the badge icon alongside the article together with a link to the disclosed data, materials, or registration.
Research Transparency
Research transparency is the practice of making the specific decisions, materials, data, and code behind a study visible and checkable at the point they are used, not the broader movement to open up research outputs generally (that is <a href='/dictionary/term/open-science'>open science</a>). A study demonstrates research transparency to the degree it discloses, and ideally shares and cites, what it did: whether hypotheses and analysis plans were specified before or after seeing the data, where materials and data can be obtained, whether analytic code is available, and whether the design and reporting follow a recognized standard. The most widely adopted operational framework for this is the Transparency and Openness Promotion (TOP) Guidelines, published by the Center for Open Science (COS), which turns 'be transparent' into eight checkable standards — citation, data transparency, analytic methods (code) transparency, research materials transparency, design and analysis transparency, study preregistration, analysis plan preregistration, and replication — each ratable at one of three levels of stringency, from disclosure through requirement to independent verification.
SCORE (Systematizing Confidence in Open Research and Evidence)
A DARPA-funded research program, run by DARPA's Defense Sciences Office from 2019 to roughly 2022, that tested whether quantitative 'confidence scores' -- generated from human forecasting/prediction markets, expert surveys, and automated algorithms -- could reliably estimate how likely a published social and behavioral science (SBS) claim was to replicate. The program sampled around 3,000 research claims from roughly 60 journals published between 2009 and 2018, had forecasters and algorithms score them before any new data collection, then commissioned independent replications on a subsample to check which forecasting method came closest to the actual outcome. It is now complete; DARPA lists it as inactive and retained for reference.
Journal Article Reporting Standards (JARS)
Journal Article Reporting Standards (JARS) is the American Psychological Association's set of manuscript-section reporting requirements, published as part of APA Style and incorporated into the <em>Publication Manual of the APA</em> (7th edition). A manuscript is JARS-compliant when the author has followed the module matching its research design: <strong>JARS-Quant</strong> for quantitative studies (numerical data, statistical analysis), <strong>JARS-Qual</strong> for qualitative studies (natural-language, descriptive, or interview-based data), or <strong>JARS-Mixed</strong> for studies combining both. A cross-cutting fourth module, <strong>JARS-Race, Ethnicity, and Culture (JARS-REC)</strong>, adds reporting expectations for how race, ethnicity, and culture are described and analyzed, and applies regardless of which core module a study uses. Each module specifies, section by section (title/abstract, introduction, method, results, discussion), the minimum information a submission must state -- for example, JARS-Quant requires reporting exact sample-size determination and any data exclusions; JARS-Qual requires describing the researcher's own standpoint and role in relation to the data. JARS is APA's own umbrella framework and is distinct from discipline- or design-specific reporting guidelines such as PRISMA (systematic reviews), CONSORT (randomized trials), or COREQ/SRQR (qualitative research specifically) -- a paper can be asked to satisfy JARS alongside one of those other checklists, not instead of it, since JARS covers general manuscript-section content while the others target a specific study design in more procedural depth.
Rigor and Transparency Index (RTI)
The Rigor and Transparency Index (RTI) is the average SciScore — an automated score of methods-rigor and resource-identification reporting — across a journal's or institution's articles in a given year, published by SciCrunch as a benchmarking metric for reporting quality. It is calculated independently of citation-based metrics such as Journal Impact Factor and is meant to be compared only within the same field or subfield.
Open Science Framework (OSF)
A free, open-source web platform operated by the nonprofit Center for Open Science (COS) that lets a research team manage a project's full lifecycle from one persistent, citable project page: file storage and version control, collaborator permissions, preregistration of hypotheses and analysis plans, and public archiving with a DOI. 'Open Science Framework' is often shortened or misremembered as 'Open Science Foundation' — the organization that builds and runs OSF is the Center for Open Science, not an 'Open Science Foundation'; no organization by that name operates this infrastructure.
Data Dredging
Searching a dataset for statistically significant relationships across many variables, subgroups, or model specifications without a hypothesis specified in advance, then reporting whichever pattern crosses a significance threshold as if it had been the study’s pre-specified object of investigation, without correcting for the number of comparisons actually performed.
SciScore
SciScore is an automated text-mining tool from SciCrunch that scores a manuscript's methods section (0-10) against named rigor and resource-transparency criteria -- randomization, blinding, sample-size justification, sex as a biological variable, and RRID-identified research resources -- drawn from NIH rigor guidelines, MDAR, ARRIVE, CONSORT, and STAR Methods, integrated into journal submission workflows at publishers such as the American Heart Association, Rockefeller University Press, and FASEB.
PRISMA Flow Diagram
The standardized figure PRISMA 2020 uses to report the flow of records through a systematic review or meta-analysis, organized under three labeled phases (Identification, Screening, Included), with a numeric box for every stage of attrition and a stated reason for every full-text exclusion.
protocols.io (concept)
protocols.io is a free, open platform (owned by Springer Nature since 2023) for developing, versioning, and sharing detailed step-by-step research method protocols across life-science, physical-science, and computational disciplines. Each published protocol receives a DataCite DOI, so a specific, citable version of a method persists even as the protocol is later revised, forked, or optimized by other labs.
PROSPERO Registration
PROSPERO registration is the prospective submission of a systematic review's protocol (research question, PICO, search strategy, eligibility criteria, and analysis plan) to the International Prospective Register of Systematic Reviews (PROSPERO), maintained by the Centre for Reviews and Dissemination (CRD) at the University of York and funded by the UK's NIHR, before the review has progressed beyond data extraction. Eligible reviews must have a health-related outcome; a registered record receives a permanent, publicly citable CRD42-prefixed identifier.
Open Science
The umbrella movement and set of practices for making every stage of the research process — publications, data, code, methods, and materials — openly accessible, transparent, and reusable, rather than treating openness as something that applies only to the final published article. UNESCO’s 2021 Recommendation on Open Science, the first international normative instrument on the subject, organizes it into four pillars: open scientific knowledge, open science infrastructures, open engagement of societal actors, and open dialogue with other knowledge systems.
Intent-to-Treat (ITT) vs. Per-Protocol Analysis
An analysis-population choice for a randomized trial: an intent-to-treat (ITT) analysis includes every participant in the treatment group to which they were originally randomized, regardless of whether they received, adhered to, or completed that assigned treatment. A per-protocol (PP) analysis instead restricts the analysis to the subset of participants who received the assigned treatment as specified in the protocol, with no major deviations. ITT answers "what happens if this treatment is assigned"; per-protocol answers "what happens if this treatment is actually taken as directed" — and which of the two gives the more conservative estimate depends on whether the trial is testing superiority or non-inferiority.
PROV-O (Provenance Ontology)
PROV-O is the W3C's OWL2 ontology (a W3C Recommendation since 30 April 2013) for expressing data provenance as machine-readable RDF: it defines Entity (a thing with fixed aspects), Activity (something that acts upon or generates entities over time), and Agent (something responsible for an activity or entity), connected by properties such as wasGeneratedBy, used, wasAssociatedWith, and wasDerivedFrom.
RRID (Research Resource Identifier)
A string of the form RRID: followed by an accession number issued by an authoritative source database (the Antibody Registry for AB_ numbers, Cellosaurus for CVCL_ cell-line codes, MGI/ZFIN/RGD/IMSR for model organisms, or SciCrunch's own SCR_ registry for software and databases) that resolves, via the SciCrunch resolver or a compatible resolver such as n2t.net, to a single curated record describing exactly one research resource. Developed through FORCE11's Resource Identification Initiative and maintained via SciCrunch's Resource Identification Portal, an RRID exists specifically so a research resource used in a study's methods can be cited unambiguously and traced across the literature, the way a DOI identifies a publication or an ORCID iD identifies a researcher.
Randomized Controlled Trial (RCT)
A prospective clinical study design in which participants are allocated to two or more comparison groups purely by chance (randomization), with at least one group — the control arm — receiving a placebo, an active comparator, or standard-of-care rather than the intervention under investigation. Randomization is what distinguishes an RCT from an observational study; the presence of a control arm is what distinguishes it from a single-arm trial. Blinding (masking) of participants, investigators, or outcome assessors is a common but analytically separate design feature — an RCT can be randomized, controlled, and still unblinded (open-label).
Metascience
The empirical, quantitative study of science itself: applying research methods to research methods, reporting practices, evaluation systems, and incentive structures, in order to describe how research is actually conducted and to test interventions that make it more reliable and efficient. Something counts as metascience when its object of study is the research process across a field — study design, statistical power, reporting completeness, peer review, replication rates, or how funding and career incentives shape researcher behavior — rather than a first-order scientific question within a specific discipline. Also called meta-research or, informally, "the science of science."
Scientific rigour
The strict application of the scientific method to ensure unbiased and well-controlled experimental design, methodology, analysis, interpretation, and reporting of results.
Authentication of key resources
The verification, by methods appropriate to the resource type, of the identity and integrity of biological and chemical materials used in research, including cell lines, antibodies, animal models, and specialty chemicals, as required by the NIH Rigor and Reproducibility policy for resources central to a proposed study.
NIH Rigor and Reproducibility policy
The set of US National Institutes of Health policies, effective from 2016, requiring applicants and grantees to address scientific premise, scientific rigour, biological variables (including sex as a biological variable), and authentication of key biological and chemical resources in grant applications.
STROBE
STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) is a 22-item reporting checklist for observational study designs — cohort, case-control, and cross-sectional — published by von Elm et al. in 2007.
PRISMA 2020
The 2020 update of the Preferred Reporting Items for Systematic reviews and Meta-Analyses, a 27-item checklist with accompanying flow diagram for reporting systematic reviews.
CONSORT 2010
The 2010 edition of the Consolidated Standards of Reporting Trials, a 25-item checklist and participant-flow diagram covering items that should be reported in any randomised controlled trial publication.
ARRIVE 2.0
ARRIVE 2.0 is the 2020 revision of the ARRIVE (Animal Research: Reporting of In Vivo Experiments) reporting guideline, restructuring the original 2010 checklist into a 10-item "Essential 10" minimum-reporting set and an 11-item "Recommended Set" for animal research manuscripts.
TOP Guidelines
The Transparency and Openness Promotion Guidelines, an eight-standard framework for journal policies covering citation, data, materials, code, design, analysis, pre-registration, and replication.
Crowdsourced replication
A coordinated effort in which many independent laboratories or teams attempt to replicate the same set of studies under pre-specified protocols, in order to estimate field-wide replicability.
Many-analysts study
A many-analysts study is a research-methods design in which multiple independent teams are given the identical dataset and research question and asked to analyse it separately, used to directly measure how much a study’s conclusions vary purely as a function of analytic choices, holding the data and question constant.
Researcher degrees of freedom
The many individually defensible decision points in designing, running, analysing and reporting a study -- exclusions, covariates, outcome measures, stopping rules -- that collectively let a researcher reach a significant result far more easily than one pre-specified analysis would allow.
Garden of forking paths
The Gelman-Loken metaphor for the implicit, data-dependent multiplicity of analytical choices made in the course of an empirical study, even by analysts not engaged in explicit p-hacking.
Forking paths
The phenomenon by which the cumulative effect of many small, data-contingent analytical choices inflates false-positive rates even when each individual choice appears defensible.
HARKing (Hypothesising After Results are Known)
HARKing is presenting a post hoc, data-driven hypothesis in a paper's Introduction as though it were the a priori hypothesis that motivated the study's design, disguising exploratory findings as confirmatory ones.
P-hacking
P-hacking is the practice of exploiting flexibility in data collection, exclusion, or analysis choices — often unconsciously — to steer a result toward statistical significance, undisclosed in the final report as if the reported analysis were the only one that had been tried.
Reproducibility audit
A systematic, post-publication examination of whether a study's published results can be obtained from its deposited data and code, typically performed by an independent analyst.
Code review (research software)
Code review for research software is a check of analysis or research software code by someone other than its author, for correctness, clarity, and reproducibility, before or alongside publication of the results it produced.
FAIR4RS (FAIR Principles for Research Software)
Software satisfies FAIR4RS to the extent it meets the four adapted FAIR categories -- Findable, Accessible, Interoperable, Reusable -- with sub-principles rewritten for software's executable, versioned, dependency-bound nature (RDA/FORCE11/ReSA, 2022, DOI 10.15497/RDA00068). It is a standard for making software FAIR, not for citing it -- see the separate FORCE11 Software Citation Principles.
Reproducible Research Practices (RRP)
Reproducible research practices are the concrete workflows a research team adopts — version-controlled code, pinned computational environments, data and code deposit, documented pipelines, pre-registration where applicable — that make a specific published result independently checkable.
Nextflow (concept)
A workflow orchestration system based on dataflow programming with a Groovy-based domain-specific language, designed for scalable, container-native, multi-platform execution of computational pipelines.
Snakemake (concept)
A Python-based workflow management system that expresses computational pipelines as rules with explicit inputs, outputs, and shell or script bodies, and infers a directed acyclic graph (DAG) of jobs from those rules.
Container image (Docker/Singularity/Apptainer)
A packaged, immutable filesystem and configuration that contains an application together with all its dependencies, runnable identically on any compatible container engine (Docker, Podman, Singularity, Apptainer). A Singularity container specifically refers to this artifact built or run via the Singularity/Apptainer engine -- a single portable .sif image file executed without a privileged daemon.
Workflow Language (CWL/WDL)
A declarative specification language for describing multi-step computational analyses such that the steps, their inputs and outputs, and their software dependencies are portable across compatible workflow execution engines.
Computational environment
The full software and hardware context in which an analysis runs, including operating system, language runtime, library versions, configuration, environment variables, and hardware-specific dependencies (e.g., GPU drivers).
Code Availability Statement
A statement in a published article describing where the source code used in the study can be obtained, under what licence, and at what version, typically required by journal policy.
Data availability statement
A data availability statement is a mandatory manuscript section, distinct from the paper's methods, that tells readers where and under what conditions the underlying research data can be obtained.
Open data
The practice of making research data freely available for any user to access, use, modify, and share, subject only to attribution requirements, typically through deposit in a public repository under an open licence.
Open materials
The release of the non-data, non-code materials used in a study (stimuli, survey instruments, experimental protocols, training materials, intervention manuals) such that future investigators can re-implement the procedure.
Open code
The practice of releasing the source code used in a study, under an open-source licence, alongside the publication, such that any reader may inspect, reuse, and re-execute the analysis.
Robustness check
An additional analysis, supplementary to the headline result, that varies one or more analytical choices in order to demonstrate that the main conclusion is not artefactual to those choices.
Multiverse analysis
An analytical approach in which all reasonable combinations of data-processing and modelling choices are executed, producing a distribution of results that displays the impact of researcher degrees of freedom on the conclusion.
Specification curve
An analytic and visual technique that plots the estimated effect across a large set of theoretically defensible model specifications, ordered by effect size, to convey the sensitivity of the result to analytical choices.
Pre-analysis plan
A detailed, time-stamped document specifying the statistical models, variable transformations, exclusion criteria, and inference rules to be applied to a dataset, lodged before the analyst sees the outcome data.
Pre-registration
The practice of publicly recording a study's hypotheses, design, sample, and analysis plan in a time-stamped registry before data collection or (in secondary-data work) before data access, in order to distinguish pre-specified from post-hoc analyses.
Reproducibility crisis
The widely reported finding that substantial proportions of published research, particularly in biomedical, psychological, and social sciences, fail to reproduce or replicate when re-tested.
Inferential reproducibility
Inferential reproducibility is when independent investigators, working from the same or comparable data, reach qualitatively similar scientific conclusions, even where their specific analytical choices differ — the highest of the three reproducibility types proposed by Goodman, Fanelli and Ioannidis (2016).
Results reproducibility
The narrow sense in which a study's reported quantitative results can be recreated from the deposited data using the deposited analysis procedures.
Methods reproducibility
The degree to which a study's methods are reported in sufficient detail that another investigator could re-implement them, independent of whether the same numerical or empirical results would follow.
Empirical reproducibility
The ability to obtain consistent observations when an empirical procedure (laboratory, field, or measurement) is independently repeated under matched conditions.
Computational reproducibility
The narrow technical sense of reproducibility: obtaining the same numerical outputs from the same data and code, on a comparable computational environment.
Generalisability
The extent to which a study's findings extend to populations, settings, or conditions other than those directly sampled.
Robustness
The property of a finding, estimate, model, or system remaining materially unchanged under defensible variation in analytic choices, input data, or operating conditions, rather than depending on one narrow configuration.
Replicability
The ability to obtain consistent results when an independent investigator collects new data using the same study design and analysis procedures.
Reproducibility
The ability to obtain consistent computational or analytical results when the same data and analysis procedures are applied by an independent investigator using the same code and tools.
70 terms beyond the curated set above.
Stewardship
Reproducibility terminology sits between three communities. The CASRAI Dictionary working group integrates and publishes the vocabulary. The Center for Open Scienceis the originating steward of the TOP Guidelines, the Registered Reports format, and the preregistration registry on the Open Science Framework. The Committee on Publication Ethics (COPE)stewards the boundary cases where non-replication intersects with editorial concern.
Cross-reference to CODATA covers the related FAIR and RDM stewardship that touches reproducibility from the data-management side. The Reproducibility working group also liaises with FORCE11 on FAIR4RS and software-citation practice, with the EQUATOR Network on reporting guidelines, and with NIH and Wellcome on funder-policy alignment.
Related editorial and standards
- Reproducibility standards hub — the standards-level companion to this domain
- CRediT for authors — how contributor attribution supports reproducibility
- CRediT role: Validation — the role most directly mapped to reproducibility checks
- CRediT role: Data Curation — the role that makes computational reproducibility possible
- FAIR principles assessment — cross-domain reference
- CASRAI × CODATA federation — adjacent RDM stewardship
Federation
Cross-walks to external standards
Each entry in this domain carries machine-readable mappings to the upstream standard. Definitions remain canonical at the steward; CASRAI federates rather than re-publishes.
| Standard | Steward | Scope |
|---|---|---|
| TOP Guidelines | Center for Open Science | Eight modular journal-policy standards covering citation, data, code, materials, design, analysis, preregistration, replication. |
| ARRIVE 2.0 | NC3Rs (du Sert, Ahluwalia, Alam et al. 2020) | Reporting checklist for in vivo animal research; Essential 10 + Recommended Set. |
| NIH Rigor & Reproducibility | US National Institutes of Health | Four-pillar policy covering scientific premise, rigorous design, biological variables, and authentication of key resources. |
| CONSORT · PRISMA · STROBE | EQUATOR Network | Reporting guidelines for trials, systematic reviews, and observational studies respectively. |
Frequently asked questions
Are reproducibility and replicability the same thing?
No. CASRAI follows the National Academies (NASEM 2019) usage. Reproducibility refers to obtaining the same result from the same data and code; replicability refers to obtaining a consistent result from a new study aimed at the same question. Robustness is a third, distinct construct concerning stability under analytic variation. Conflating the three is the single most common source of definitional drift in the literature, which is why the Dictionary maintains them as separate entries.
Why does CASRAI maintain reproducibility terminology if NIH and COS already do?
NIH defines reproducibility in the context of US biomedical grant policy; COS defines it in the context of TOP-compliant journal policy. Neither is a vendor-neutral controlled vocabulary that a CRIS, repository, or publisher submission system can ingest as machine-readable terms. CASRAI integrates these definitions, federates with their stewards, and exposes the result as Schema.org DefinedTerm markup under CC-BY 4.0.
Where do I propose a new reproducibility term?
Use the contribute flow. The Reproducibility working group reviews proposals during each release cycle (March and September). Accepted contributors receive CRediT attribution on the term entry itself.
How does CRediT support reproducibility?
CRediT provides the contributor-role granularity that reproducibility audits depend on. The Validation role in particular maps directly to reproducibility checks; Data Curation and Software map to the artefacts that make computational reproducibility possible. The Dictionary cross-references CRediT roles from every relevant reproducibility term.
Related across CASRAI
How reproducibility connects to contributor attribution, the research-administration workflow, and the wider CASRAI standards.
