Under DORA and the CoARA Agreement, quantitative indicators such as the Journal Impact Factor and h-index must never substitute for expert peer judgement in hiring and promotion decisions — they may only inform it, applied with clarity, transparency, specificity, context and fairness, alongside a broader account of a candidate’s contributions.
Quantitative indicators in research assessment are numerical proxies for research activity — citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetrics — used, under explicit caveats, to inform rather than replace qualitative evaluation of a researcher’s work.
Research offices translating this principle into a hiring or promotion brief face a harder question than “which metrics are banned?” Panels need operational wording for the call, the assessor briefing and the case file. This guide sets out what DORA, the CoARA Agreement and the UK’s Forum for Responsible Research Metrics concretely require, and how to turn that into panel-ready criteria.
Contents
- What counts as a quantitative indicator?
- What does DORA require for hiring and promotion panels?
- What does the CoARA Agreement commit panels to?
- Translating principles into concrete panel criteria
- Frequently asked questions
- Implications for research offices
What counts as a quantitative indicator in research assessment?
A quantitative indicator is any numerical measure derived from research outputs or activity: citation counts, the h-index, the Journal Impact Factor (JIF), field-normalised citation ratios, grant income, patent counts and altmetric mentions all qualify. None was designed to certify the quality of a single article or a single person’s contribution.
The University of York’s policy for research evaluation using quantitative data, approved by its Research Committee in November 2017, makes the distinction explicit: indicators are informative at departmental or institutional level, but “the assessment of individual research performance using solely quantitative indicators is not supported.” That collective-versus-individual distinction is the fault line every hiring and promotion policy has to draw.
What does DORA require for hiring and promotion panels?
The San Francisco Declaration on Research Assessment (DORA), agreed in December 2012, states a single unambiguous prohibition that panels must apply: do not use journal-based metrics, such as the Journal Impact Factor, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions. That sentence, not a general suspicion of numbers, is DORA’s operative rule for panels.
DORA does not ban quantitative indicators outright. Its 2024 guidance document on the responsible use of quantitative indicators, produced by a DORA task force chaired by Professor Stephen Curry and published via Zenodo, sets out five principles that must govern any indicator a panel does choose to use: be clear, be transparent, be specific, be contextual, and be fair. These are DORA’s own words. Some AI-generated summaries currently paraphrase this as “the five Cs” — clarity, context, calibration, care, credit — a mnemonic that does not appear anywhere in DORA’s published guidance; panels drafting criteria should cite DORA’s actual five principles instead.
Applied to a panel: state which indicator is being consulted and why (clear); disclose the data source and calculation method (transparent); tie the indicator to the specific claim it supports, not a general quality judgement (specific); benchmark against discipline and career stage (contextual); and check for bias against gender, geography, career breaks or non-traditional outputs (fair).
What does the CoARA Agreement commit panels to?
The Coalition for Advancing Research Assessment (CoARA) launched its Agreement on Reforming Research Assessment in 2022, since signed by several hundred universities, funders, national agencies and learned societies across Europe and beyond. The Agreement sets ten commitments; its core, non-negotiable commitment is to “abandon inappropriate uses in research assessment of journal- and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index.”
Beyond that prohibition, CoARA’s commitments push panels toward qualitative peer review as the primary method, recognition of a wider range of outputs — datasets, software, protocols, policy engagement, mentoring and open-science practice — and narrative formats such as narrative CVs that let candidates describe contributions in their own words.
The table below compares the three frameworks a UK or European research office is most likely to be asked to reconcile.
| Framework | Origin and scope | Core requirement for hiring/promotion | Status of quantitative indicators |
|---|---|---|---|
| DORA | Global; agreed San Francisco, December 2012 | Do not use JIF as a surrogate for individual quality in hiring, promotion or funding decisions | Conditional use only, governed by five principles: clear, transparent, specific, contextual, fair |
| CoARA Agreement | Pan-European coalition; launched 2022 | Core commitment to abandon inappropriate JIF/h-index use in individual assessment | Indicators permitted only to support, not replace, qualitative peer review |
| Forum for Responsible Research Metrics (UK) | UK sector body, stemming from The Metric Tide (Wilsdon et al., HEFCE, 2015) | Institutions asked to publish a responsible-metrics statement covering hiring/promotion criteria | Five dimensions: robustness, humility, transparency, diversity, reflexivity |
Translating principles into concrete panel criteria
Principles do not write themselves into a job description. A defensible panel criteria set, translating DORA, CoARA and Forum for Responsible Research Metrics guidance into working practice, includes:
- State in the call and case-file template that the Journal Impact Factor, h-index and journal rank will not be used as proxies for individual quality (DORA’s core recommendation).
- Offer or require a narrative CV alongside, or instead of, a conventional publication list, so data, software, mentoring and open-science contributions are visible to assessors.
- If citation data is used at all, require field-normalised indicators rather than raw counts, and disclose the source database in the case file.
- Credit non-publication outputs explicitly in the assessment rubric, consistent with CoARA’s broadened-recognition commitments.
- Brief panel members on indicator limitations before each cycle, per the Forum for Responsible Research Metrics’ “humility” dimension.
- Record, for each case, which indicators (if any) were consulted and the specific claim they supported (DORA’s “transparent” and “specific” principles).
- Review the criteria annually, reflecting the “reflexivity” dimension shared by the Leiden Manifesto (Hicks et al., Nature, 2015) and the Forum for Responsible Research Metrics.
A useful complementary vocabulary for the “credit non-publication outputs” step is a structured contributor-role taxonomy. CASRAI originated the CRediT contributor role taxonomy in 2014; the standard is now stewarded by NISO as ANSI/NISO Z39.104-2022. Panels reviewing narrative CVs can use CRediT’s fourteen roles to make specific, verifiable contribution claims — distinguishing data curation from formal analysis, for example — rather than relying on author order or citation counts as a proxy for who did what.
Frequently asked questions
What are quantitative indicators in research assessment?
Quantitative indicators are numerical measures of research activity, including citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetric mentions. DORA’s guidance treats them as descriptive data points requiring context, not standalone quality scores, and warns against using any single indicator in isolation.
Does DORA allow any use of quantitative indicators in hiring and promotion?
Yes, conditionally. DORA does not ban indicators outright; it prohibits journal-based metrics like the Journal Impact Factor as a surrogate for individual quality in hiring, promotion or funding decisions. Where indicators are used, DORA’s five principles — clear, transparent, specific, contextual, fair — must govern their application.
What does the CoARA Agreement require of hiring and promotion panels?
CoARA’s core commitment obliges signatories to abandon inappropriate use of journal- and publication-based metrics, particularly the Journal Impact Factor and h-index, in individual assessment. Panels must prioritise qualitative peer judgement, broaden recognised output types, and adopt formats such as narrative CVs.
What is a narrative CV, and is it required under responsible metrics guidance?
A narrative CV lets candidates describe significant contributions — including data, software, mentoring and open-science practice — in their own words, rather than through a publication-and-citation list. DORA and CoARA both recommend narrative formats to support qualitative review, though neither makes them a formal, binding requirement.
Implications for research offices
UK institutions face a specific reconciliation problem: government is considering bibliometric data as an optional component of the next Research Excellence Framework exercise, REF 2029, at discipline and institutional level, even as DORA and CoARA prohibit citation-based proxies at the level of the individual hire. Policy wording needs to keep these two scales distinct — permitting aggregate bibliometric reporting upward to funders while barring the same data from an individual case file.
The direction of travel across DORA and CoARA signatories is consistent: fewer single-number thresholds, more disclosed and contextualised indicator use, and a growing expectation that panels can explain, in writing, which evidence supported which judgement. Research offices that build this documentation habit now, rather than waiting for a funder or auditor to ask, will find each subsequent cycle easier to defend, not harder.
