CASRAI Dictionary

Tag: registered report

  • Registered reports and pre-registration: planning research in the open

    Most research is reviewed and published after the results are known. That ordering, so obvious it usually goes unexamined, quietly distorts the literature: studies with striking positive results get published and studies with null results get filed away, and analyses can drift, after the fact, toward whatever story the data happen to tell. Pre-registration and its more rigorous cousin, the registered report, flip the order — committing to the plan before the data exist — and in doing so address some of the deepest threats to reproducibility. They are a central concern of the reproducibility domain and connect directly to the research-integrity domain.

    The problems they are designed to solve

    Two well-documented distortions motivate planning research in the open.

    The first is publication bias: the tendency for positive, “significant” results to be published while null or negative results disappear. The literature that results is not a fair sample of the research that was done — it over-represents flukes and under-represents the disconfirmations that science depends on. A field can end up confidently believing an effect that the full body of evidence, published and unpublished, would not support.

    The second is the family of analytic flexibility problems, of which HARKing — Hypothesising After the Results are Known — is the clearest example. When the hypothesis is written after seeing the data, and when there is freedom to choose among many possible analyses, it becomes easy, often unintentionally, to present an exploratory finding as if it had been predicted, and to select the analysis that produces the most publishable result. None of this need involve any intent to deceive; it is the natural consequence of making decisions while looking at the outcome.

    Pre-registration: committing to the plan

    Pre-registration is the practice of specifying, in a public, time-stamped record before data collection or analysis, what the study will do: its hypotheses, its design, its sampling and stopping rules, its outcome measures, and its planned analysis. The record is created in advance and cannot be quietly altered afterwards, which draws a clean line between what was confirmatory (predicted in advance) and what was exploratory (discovered in the data). Exploratory analysis remains entirely legitimate and valuable — pre-registration does not forbid it; it simply makes it honest by preventing exploratory findings from being dressed up as confirmatory ones.

    The Open Science Framework (OSF), maintained by the non-profit Center for Open Science, is the most widely used infrastructure for this. OSF lets researchers create a registration — a frozen, time-stamped, citable snapshot of the study plan — and control when it becomes public. The plan is fixed; the credibility of any later claim to have predicted a result can be checked against it.

    Registered reports: review before the results

    A registered report takes the logic further and builds it into the publishing process itself, through a two-stage peer review designed and promoted by the Center for Open Science and now offered by a large and growing number of journals.

    • Stage 1 is the protocol. Before any data are collected, the authors submit the introduction, the hypotheses, and a detailed methods and analysis plan. Reviewers assess the importance of the question and the soundness of the method — not the results, because there are none yet. If the protocol passes, the journal grants in-principle acceptance: a commitment to publish the completed study regardless of how the results turn out, provided the authors carry out the registered plan and the work is sound.
    • Stage 2 is the completed study. The authors execute the plan, report what they found — positive, null, or mixed — clearly distinguish any exploratory analyses from the pre-registered confirmatory ones, and the paper is published.

    The consequences are precise. Because the decision to publish is made before the results are known, publication bias is removed at its source — a null result is just as publishable as a positive one. Because the analysis plan is fixed and reviewed up front, HARKing and selective analysis are structurally prevented. And because reviewers shape the design while it can still be improved, peer review does its most useful work before the study is run rather than after, when nothing can be changed.

    What this strengthens, and what it does not

    Registered reports and pre-registration are powerful but not universal. They suit hypothesis-testing, confirmatory research best; they fit awkwardly onto genuinely exploratory, discovery-driven, or qualitative work, where the questions emerge from the material and a rigid pre-specified plan would be a forced fit. The honest position is that they are an excellent tool for a particular and very common kind of research, not a mandate for all of it. Used where they fit, they directly serve reproducibility: a study whose plan was fixed and public in advance is far easier for others to evaluate, replicate, and build on.

    Crediting the planning work

    Planning a study rigorously is itself a substantial contribution, and contributor-role metadata can record it. The CRediT taxonomy‘s Conceptualization and Methodology roles capture the intellectual work of formulating the research goals and designing the methods — precisely the work that a registered report front-loads and makes visible. Recording these roles ensures that the design effort, which a registered report elevates from invisible preparation to peer-reviewed output, is credited to the people who did it.

    Where shared vocabulary fits

    “Pre-registration”, “registered report”, “in-principle acceptance”, “Stage 1 protocol”, and “confirmatory analysis” are used loosely and sometimes interchangeably, which muddies what a given journal or record actually guarantees. A shared, federated vocabulary that defines these terms precisely — and points back to the Center for Open Science and the OSF registration infrastructure — is what lets a registered report in one venue be understood the same way in another. Supplying that definitional layer is the role the CASRAI dictionary is designed to play; the relevant terms sit in the reproducibility domain, with adjacent entries in the research-integrity domain.

    Related reading

  • Beyond the article: a modern taxonomy of research outputs

    For most of the history of the scholarly record, “research output” meant one thing: the peer-reviewed journal article, with the book a distant second. Everything else — the data, the code, the protocols, the negative results — was apparatus or supplement, uncounted and largely uncredited. That assumption has broken down, and a modern outputs taxonomy has to reflect a far wider range of things that researchers produce, each deserving its own place, its own identifier, and its own recognition in assessment. This article surveys that expanded taxonomy, drawing on the research-outputs domain.

    Why the article-only model failed

    The article-centric model failed for a simple reason: the article is no longer where much of the value lives. A reproducible computational study’s value is as much in its code and data as in its prose. A widely reused dataset can influence a field more than the paper that introduced it. A protocol followed by hundreds of labs is a contribution in its own right. Treating all of these as mere supplements to an article misallocates credit, loses the artefacts that actually get reused, and gives assessment a distorted picture of what a researcher contributed. The expansion of the outputs taxonomy is not taxonomic enthusiasm; it is a correction.

    The expanded output types

    A modern taxonomy organises a wide range of outputs. Several stand out as having reshaped the landscape.

    • The preprint — a manuscript posted to a public server before or during formal peer review — is now a first-class output, not a second-class draft. It establishes priority, accelerates dissemination, and carries its own DOI. The relationship between a preprint and its eventual published version is itself metadata worth recording.
    • The dataset — a collection of research data with a DataCite DOI — is the output whose recognition has changed most. Data citation is now expected practice, and a well-curated, documented dataset is a citable contribution that can be credited and assessed.
    • Research software — software produced for or as part of research, with a stable identifier such as a Software Heritage ID or a DataCite DOI — is increasingly recognised as a research output, with its own citation conventions and its own (imperfect) fit to contributorship taxonomies.
    • The trained model — an AI/ML model released as a research output, typically documented with a model card — is the newest major addition, reflecting the rise of machine-learning research that produces models and datasets rather than only papers.
    • The registered report — published in two stages, with the protocol peer-reviewed and accepted before data collection — is a structural innovation in how an output is produced, designed to guard against publication bias by committing to publish regardless of outcome.

    Beyond these, the taxonomy reaches further: protocols with DOIs (as minted on platforms like protocols.io), negative-results reports, systematic reviews and their living variants, policy briefs, standards contributions, patents, clinical-trial registrations, theses, conference papers, and the practice-based outputs of the arts. The breadth is the point: research produces many kinds of thing, and a taxonomy that names only one of them is misleading by omission.

    Two structural requirements: identifiers and relationships

    An outputs taxonomy is only useful if its entries can be reliably identified and related. Two requirements follow.

    The first is stable identifiers for every output type, not just articles. A dataset needs a DOI, software needs a SWHID or DOI, a sample referenced by an output needs an IGSN, a project that produced the outputs needs a RAiD, and the people and institutions need ORCID and ROR. Without identifiers, the expanded taxonomy is just a longer list of things that cannot be cited or counted reliably. With them, every output type becomes a first-class, citable, assessable entity.

    The second is clean parent-child and related relationships between output types. A registered report’s stage-1 protocol and stage-2 article are related; a preprint and its published version are related; a dataset and the software that processed it are related; a systematic review and the studies it synthesises are related. A taxonomy that captures these relationships lets automated systems and CRIS platforms reason over outputs — grouping a project’s preprint, dataset, and software as facets of one contribution rather than three unconnected records.

    Why this matters for assessment

    The expanded taxonomy connects directly to responsible research assessment. Narrative-CV formats explicitly invite researchers to describe contributions beyond publications — the datasets, the software, the open-science work. But for an assessor to take a dataset or a model seriously, it has to be a recognised, identifiable output type, not an undifferentiated “other.” A modern outputs taxonomy is the precondition for assessment that values what researchers actually produce. Naming a model, a dataset, or a protocol as a first-class output is what lets it be claimed on a CV and weighed by a panel.

    A caution against type proliferation

    A taxonomy can fail in two directions. The old failure was too few types — everything that was not an article was invisible. The opposite failure is too many: a sprawling list of hyper-specific types that no two systems classify the same way, so that exchange becomes impossible and the taxonomy collapses under its own weight. The discipline a good taxonomy needs is to enumerate the types that genuinely behave differently — that have different identifiers, different lifecycles, different assessment treatment — and to use relationships rather than ever-finer types to capture the rest. The goal is a taxonomy that classifiers and CRIS systems can apply consistently, which means stable, well-bounded types with clean relationships, not an open-ended catalogue.

    Where the dictionary fits

    Several stewards already maintain output-type vocabularies — COAR Resource Types, the Crossref and DataCite output types, the categories used by national assessment exercises. The need is not another competing list but an integrative, operational reference that defines each type clearly, federates to those stewards, and makes the relationships between types explicit. Providing that — so that a “dataset” or a “registered report” means the same thing across systems — is the convening role the CASRAI dictionary is designed for.

    What to do now

    For researchers: mint identifiers for all your outputs, not only your papers, and record the relationships between them. For institutions and CRIS owners: support the full range of output types as first-class records with clean relationships, federating your type list to an established vocabulary rather than inventing one. For assessment: recognise the expanded taxonomy, so that the dataset, the model, and the protocol can be claimed and weighed alongside the article.

    Related reading