The movement to reform research assessment has produced a powerful set of principles. Declarations and manifestos have told the community what to stop doing: stop using journal-based metrics as a proxy for the quality of individual articles, stop reducing complex contributions to a single number, stop letting convenient indicators substitute for judgement. These principles are essential, but they leave a practical gap. An evaluator — a panel chair, a research manager, a committee designing a hiring process — who agrees with all of them still has to design and run an actual evaluation, and “don’t do the bad things” is not, by itself, a method. The SCOPE framework, developed within INORMS (the International Network of Research Management Societies), exists to fill that gap by offering a structured process for designing a responsible evaluation. This article explains it, drawing on the responsible assessment domain of the CASRAI Dictionary.
From principles to process
The distinctive contribution of SCOPE is that it is a how-to, not a what-not-to. Where DORA and the Leiden Manifesto articulate the values and warn against the failure modes of assessment, SCOPE provides a sequence of steps an evaluator can actually follow to build an evaluation that honours those values. It treats the design of an evaluation as a deliberate act requiring thought, rather than a default to be reached for unreflectively. The name SCOPE is an acronym for the stages of that process, and working through them in order is meant to prevent the most common error in assessment: choosing the measure first — usually whatever is easy to count — and only afterwards, if at all, asking whether it actually captures what matters.
The five steps
SCOPE guides an evaluator through five stages:
- S — Start with what you value. Before anything is measured, articulate what the evaluation is genuinely meant to recognise and encourage. This puts values, not available data, in the driving seat, and forces clarity about the purpose of the exercise.
- C — Context considerations. Take account of the specific context: who or what is being evaluated, the discipline, the career stage, the conditions, and the consequences the evaluation will have. An approach appropriate in one context may be unfair or meaningless in another.
- O — Options for evaluating. Consider the range of possible ways to conduct the evaluation — qualitative and quantitative, expert judgement and indicators — rather than defaulting to the most familiar tool. This is where the evaluator deliberately weighs alternatives.
- P — Probe deeply. Interrogate the chosen approach. What are its limitations, biases and unintended effects? Who might it disadvantage? What behaviour will it incentivise? Probing before committing is how harms are caught in advance.
- E — Evaluate your evaluation. After the exercise, assess whether the evaluation actually worked — whether it served its purpose, was fair, and had the intended effects — and feed what is learned back into future practice.
The order is the point. By beginning with values and context and treating measurement as a later, considered choice, SCOPE structurally resists the temptation to let convenient metrics define what counts.
How SCOPE relates to DORA, CoARA and the Leiden Manifesto
SCOPE does not compete with the major assessment-reform initiatives; it operationalises them. The San Francisco Declaration on Research Assessment (DORA) sets out commitments to stop misusing journal-based metrics and to assess research on its own merits; SCOPE gives an evaluator a way to design assessments that actually deliver on those commitments. The Leiden Manifesto offers principles for the responsible use of metrics — supporting rather than supplanting expert judgement, accounting for context, recognising the limits of indicators — and SCOPE’s steps are, in effect, a procedure for honouring those principles in a concrete exercise. The Coalition for Advancing Research Assessment (CoARA) commits its many signatory organisations to reforming how they assess research; SCOPE is precisely the kind of practical tool such organisations need to translate their commitments into the design of real evaluations. In short, the declarations supply the why and the constraints; SCOPE supplies a disciplined way to do the work within them.
Why a process matters
It is worth dwelling on why a process, rather than a set of rules, is the right form for this. Research is too varied for a single prescribed method to fit every case: what is fair when assessing a senior researcher differs from what is fair for an early-career one; what makes sense in a laboratory discipline differs from a field where books and long-form scholarship dominate. A rigid rule (“always use X”) would simply replace one bad default with another. A process like SCOPE instead equips the evaluator to make a good, context-sensitive decision each time, while guarding against the predictable failure modes. It respects the irreducible role of judgement in assessment while ensuring that judgement is exercised thoughtfully and transparently rather than by reflex.
Describing contribution for fairer assessment
Responsible evaluation depends on having good information about what people have actually contributed, described in a way that does not collapse into crude proxies. This is where structured contribution information supports the goals of frameworks like SCOPE. The CRediT taxonomy — with its full set of contribution roles — lets an evaluation recognise the specific roles a person played rather than inferring contribution from authorship position or counting papers. Richer, structured information about contribution gives evaluators better material to exercise the considered judgement SCOPE is designed to support, and complements the narrative approaches increasingly used in responsible assessment. The institutional work of putting such practices in place is part of the broader remit of research administration.
A consistent foundation for evaluation
For responsible evaluation to work across institutions and systems, the information it draws on must be described consistently — contributions, outputs, roles and the rest. That consistency is what the CASRAI Dictionary provides: a shared vocabulary so that the evidence feeding an evaluation means the same thing wherever it comes from. SCOPE reminds us that good assessment is something you design, not something you default into; a shared vocabulary helps ensure the materials you design with are sound.