CASRAI Dictionary

Tag: hiring and promotion criteria

  • Quantitative Indicators in Research Assessment: A Hiring and Promotion Panel Guide

    Under DORA and the CoARA Agreement, quantitative indicators such as the Journal Impact Factor and h-index must never substitute for expert peer judgement in hiring and promotion decisions — they may only inform it, applied with clarity, transparency, specificity, context and fairness, alongside a broader account of a candidate’s contributions.

    Quantitative indicators in research assessment are numerical proxies for research activity — citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetrics — used, under explicit caveats, to inform rather than replace qualitative evaluation of a researcher’s work.

    Research offices translating this principle into a hiring or promotion brief face a harder question than “which metrics are banned?” Panels need operational wording for the call, the assessor briefing and the case file. This guide sets out what DORA, the CoARA Agreement and the UK’s Forum for Responsible Research Metrics concretely require, and how to turn that into panel-ready criteria.

    Contents

    What counts as a quantitative indicator in research assessment?

    A quantitative indicator is any numerical measure derived from research outputs or activity: citation counts, the h-index, the Journal Impact Factor (JIF), field-normalised citation ratios, grant income, patent counts and altmetric mentions all qualify. None was designed to certify the quality of a single article or a single person’s contribution.

    The University of York’s policy for research evaluation using quantitative data, approved by its Research Committee in November 2017, makes the distinction explicit: indicators are informative at departmental or institutional level, but “the assessment of individual research performance using solely quantitative indicators is not supported.” That collective-versus-individual distinction is the fault line every hiring and promotion policy has to draw.

    What does DORA require for hiring and promotion panels?

    The San Francisco Declaration on Research Assessment (DORA), agreed in December 2012, states a single unambiguous prohibition that panels must apply: do not use journal-based metrics, such as the Journal Impact Factor, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions. That sentence, not a general suspicion of numbers, is DORA’s operative rule for panels.

    DORA does not ban quantitative indicators outright. Its 2024 guidance document on the responsible use of quantitative indicators, produced by a DORA task force chaired by Professor Stephen Curry and published via Zenodo, sets out five principles that must govern any indicator a panel does choose to use: be clear, be transparent, be specific, be contextual, and be fair. These are DORA’s own words. Some AI-generated summaries currently paraphrase this as “the five Cs” — clarity, context, calibration, care, credit — a mnemonic that does not appear anywhere in DORA’s published guidance; panels drafting criteria should cite DORA’s actual five principles instead.

    Applied to a panel: state which indicator is being consulted and why (clear); disclose the data source and calculation method (transparent); tie the indicator to the specific claim it supports, not a general quality judgement (specific); benchmark against discipline and career stage (contextual); and check for bias against gender, geography, career breaks or non-traditional outputs (fair).

    What does the CoARA Agreement commit panels to?

    The Coalition for Advancing Research Assessment (CoARA) launched its Agreement on Reforming Research Assessment in 2022, since signed by several hundred universities, funders, national agencies and learned societies across Europe and beyond. The Agreement sets ten commitments; its core, non-negotiable commitment is to “abandon inappropriate uses in research assessment of journal- and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index.”

    Beyond that prohibition, CoARA’s commitments push panels toward qualitative peer review as the primary method, recognition of a wider range of outputs — datasets, software, protocols, policy engagement, mentoring and open-science practice — and narrative formats such as narrative CVs that let candidates describe contributions in their own words.

    The table below compares the three frameworks a UK or European research office is most likely to be asked to reconcile.

    Framework Origin and scope Core requirement for hiring/promotion Status of quantitative indicators
    DORA Global; agreed San Francisco, December 2012 Do not use JIF as a surrogate for individual quality in hiring, promotion or funding decisions Conditional use only, governed by five principles: clear, transparent, specific, contextual, fair
    CoARA Agreement Pan-European coalition; launched 2022 Core commitment to abandon inappropriate JIF/h-index use in individual assessment Indicators permitted only to support, not replace, qualitative peer review
    Forum for Responsible Research Metrics (UK) UK sector body, stemming from The Metric Tide (Wilsdon et al., HEFCE, 2015) Institutions asked to publish a responsible-metrics statement covering hiring/promotion criteria Five dimensions: robustness, humility, transparency, diversity, reflexivity

    Translating principles into concrete panel criteria

    Principles do not write themselves into a job description. A defensible panel criteria set, translating DORA, CoARA and Forum for Responsible Research Metrics guidance into working practice, includes:

    • State in the call and case-file template that the Journal Impact Factor, h-index and journal rank will not be used as proxies for individual quality (DORA’s core recommendation).
    • Offer or require a narrative CV alongside, or instead of, a conventional publication list, so data, software, mentoring and open-science contributions are visible to assessors.
    • If citation data is used at all, require field-normalised indicators rather than raw counts, and disclose the source database in the case file.
    • Credit non-publication outputs explicitly in the assessment rubric, consistent with CoARA’s broadened-recognition commitments.
    • Brief panel members on indicator limitations before each cycle, per the Forum for Responsible Research Metrics’ “humility” dimension.
    • Record, for each case, which indicators (if any) were consulted and the specific claim they supported (DORA’s “transparent” and “specific” principles).
    • Review the criteria annually, reflecting the “reflexivity” dimension shared by the Leiden Manifesto (Hicks et al., Nature, 2015) and the Forum for Responsible Research Metrics.

    A useful complementary vocabulary for the “credit non-publication outputs” step is a structured contributor-role taxonomy. CASRAI originated the CRediT contributor role taxonomy in 2014; the standard is now stewarded by NISO as ANSI/NISO Z39.104-2022. Panels reviewing narrative CVs can use CRediT’s fourteen roles to make specific, verifiable contribution claims — distinguishing data curation from formal analysis, for example — rather than relying on author order or citation counts as a proxy for who did what.

    Frequently asked questions

    What are quantitative indicators in research assessment?

    Quantitative indicators are numerical measures of research activity, including citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetric mentions. DORA’s guidance treats them as descriptive data points requiring context, not standalone quality scores, and warns against using any single indicator in isolation.

    Does DORA allow any use of quantitative indicators in hiring and promotion?

    Yes, conditionally. DORA does not ban indicators outright; it prohibits journal-based metrics like the Journal Impact Factor as a surrogate for individual quality in hiring, promotion or funding decisions. Where indicators are used, DORA’s five principles — clear, transparent, specific, contextual, fair — must govern their application.

    What does the CoARA Agreement require of hiring and promotion panels?

    CoARA’s core commitment obliges signatories to abandon inappropriate use of journal- and publication-based metrics, particularly the Journal Impact Factor and h-index, in individual assessment. Panels must prioritise qualitative peer judgement, broaden recognised output types, and adopt formats such as narrative CVs.

    What is a narrative CV, and is it required under responsible metrics guidance?

    A narrative CV lets candidates describe significant contributions — including data, software, mentoring and open-science practice — in their own words, rather than through a publication-and-citation list. DORA and CoARA both recommend narrative formats to support qualitative review, though neither makes them a formal, binding requirement.

    Implications for research offices

    UK institutions face a specific reconciliation problem: government is considering bibliometric data as an optional component of the next Research Excellence Framework exercise, REF 2029, at discipline and institutional level, even as DORA and CoARA prohibit citation-based proxies at the level of the individual hire. Policy wording needs to keep these two scales distinct — permitting aggregate bibliometric reporting upward to funders while barring the same data from an individual case file.

    The direction of travel across DORA and CoARA signatories is consistent: fewer single-number thresholds, more disclosed and contextualised indicator use, and a growing expectation that panels can explain, in writing, which evidence supported which judgement. Research offices that build this documentation habit now, rather than waiting for a funder or auditor to ask, will find each subsequent cycle easier to defend, not harder.

  • Research Assessment Reform: Why Collective Action Beats Solo Signatories

    Research assessment reform needs collective action because hiring, promotion and funding criteria are set independently by thousands of institutions — a single university dropping journal-based metrics gains nothing if every competing institution, funder and publisher still rewards them. Recent research-on-research literature frames this explicitly as a collective action problem: individual declarations such as DORA signal intent, but only coordinated, system-wide commitments — the model CoARA is built around — actually rewrite the incentives that determine careers.

    A collective action problem in research assessment is a situation where no single institution can achieve reform on its own without risking a competitive disadvantage, so change only happens when many actors move together under a shared, verifiable commitment.

    What Is the Collective Action Problem in Research Assessment Reform?

    A 2025 study in Minerva by sociologist Alexander Rushforth, “Research Assessment Reform as Collective Action Problem,” argues that research evaluation change cannot be reduced to individual institutional choice. Rushforth traces this through the Netherlands’ national “Recognition and Rewards” initiative, formally launched in 2019 to coordinate system-wide changes in assessment practice across the Dutch science system.

    The framing matters because it shifts the diagnosis. If assessment culture were simply a matter of institutional willpower, a DORA signature would be sufficient. If it is instead a coordination failure — where no actor can safely move first — then reform requires simultaneous, mutually reinforcing commitments from institutions, funders and publishers together.

    Why Doesn’t an Individual DORA Signature Change Hiring Criteria?

    The San Francisco Declaration on Research Assessment (DORA), launched in 2012, asks signatories to stop using journal-based metrics such as the Journal Impact Factor as a proxy for the quality of individual articles or researchers. Signing carries no binding enforcement mechanism, and DORA itself has long acknowledged that the harder work begins after signature — its 2019 guidance “You’ve signed DORA, now what?” explicitly frames hiring, promotion and funding criteria as the next, unfinished step.

    Two structural problems keep that step unfinished when institutions act alone:

    • First-mover risk. An institution that stops counting journal prestige in tenure review can be undercut in recruitment and rankings by peers who have not changed, because researcher CVs are still read against metric-based expectations elsewhere.
    • Interoperability failure. Where assessment criteria diverge sharply between institutions and countries, researcher mobility suffers — a candidate assessed holistically at one university may be filtered out by a metrics-based shortlist at the next.

    Neither problem is solved by any single signature. Both require peer institutions, funders and disciplinary societies to move on a broadly shared timetable.

    How Does CoARA’s Coordinated Model Differ From Individual Declarations?

    The Coalition for Advancing Research Assessment (CoARA) was formed around the Agreement on Reforming Research Assessment, which the European Commission signed and endorsed alongside DORA on 8 November 2022. Unlike a one-off declaration, CoARA requires member organisations to commit to a shared action plan with defined milestones, reported progress and working groups that develop common tools and criteria across institutions — moving assessment reform from individual pledge to managed, collective process.

    That coordination logic was reinforced on 4 December 2025, when CoARA and DORA released a joint statement on aligning their respective reform efforts rather than running parallel, uncoordinated campaigns. Science Europe’s April 2026 position statement, “Connecting Open Science and Research Assessment Reform,” makes the same point from the funder side: it treats open science and assessment reform as “mutually reinforcing and interdependent drivers of research cultures,” explicitly a multi-actor framing rather than an institution-by-institution one.

    Dimension Individual DORA signature Coordinated (CoARA-style) commitment
    Enforcement None — declaration of intent only Action plan with milestones and reporting
    Hiring/promotion criteria Left to each institution’s own timetable Shared working groups developing common criteria
    Competitive risk to first movers High — one institution changes alone Reduced — peers move on a shared timetable
    Researcher mobility Fragmented across institutions/countries Greater interoperability of criteria sought

    What Does the Dutch “Recognition and Rewards” Case Show?

    Rushforth’s analysis of Recognition and Rewards found that the initiative succeeded in uniting support from multiple influential national stakeholders — universities, funders and academic hospitals moving together — precisely because it was designed as a coordinated, system-wide commitment rather than a set of separate institutional pledges. It also documents genuine friction: critics raised concerns about the Netherlands “going it alone” internationally, illustrating that collective action problems exist at more than one level simultaneously — within a national system, and between that system and the rest of the world.

    The OECD’s April 2026 report “Reforming Research Assessment for Better Science” reaches a parallel conclusion at the international level, describing the current reform landscape as “a collective of organisations committed to reforming the assessment of research, researchers, and research organisations” — language that treats coordination, not individual compliance, as the operative unit of change.

    Frequently Asked Questions

    Does Signing DORA Actually Change University Hiring Practices?

    Not by itself. DORA’s own post-signature guidance states that hiring, promotion and funding decisions require separate, deliberate policy changes after signature. A signature is a public commitment; rewritten criteria documents, reviewed by hiring and promotion committees, are the actual evidence of change.

    What Is CoARA and How Does It Differ From DORA?

    CoARA is a coalition of research funders, institutions, and organisations built around the 2022 Agreement on Reforming Research Assessment. Unlike DORA’s single declaration, CoARA members commit to shared action plans, working groups and reported milestones — a coordination structure rather than a one-time pledge.

    Why Is Research Assessment Reform Described as a Collective Action Problem?

    Because no institution can safely change its own assessment criteria in isolation without risking a competitive disadvantage in recruitment and rankings. Research-on-research literature, including Rushforth’s 2025 Minerva study, argues reform requires simultaneous, coordinated commitments across many independent actors.

    Can One University Move Away From Metrics Without Being Disadvantaged?

    It can, but the Netherlands’ Recognition and Rewards case shows even a coordinated national effort faced criticism for “going it alone” relative to the rest of the world. A single institution acting without peer, funder and publisher alignment faces materially higher exposure to that same risk.

    What Should Institutions Actually Do Together?

    For research administration teams, the practical implication of the collective-action framing is direct: a DORA or CoARA signature belongs on a compliance checklist next to, not instead of, three coordination-dependent actions.

    1. Confirm hiring and promotion criteria documents have actually been rewritten, not merely a signature logged in a registry.
    2. Compare criteria against peer institutions in the same discipline and country to identify where first-mover risk is concentrated.
    3. Engage through CoARA working groups or equivalent sector bodies (ARMA, EARMA, INORMS) rather than drafting new criteria in isolation.

    Reform that stops at the signature stage produces a compliance artefact, not a changed incentive structure. The evidence from both the Dutch national case and the CoARA-DORA coordination model points the same way: assessment reform moves at the speed of the slowest coordinated group, not the fastest individual signatory. Institutions that treat their own criteria rewrite as contingent on parallel movement by peers, funders and publishers are following the pattern the research-on-research literature identifies as actually working — treating reform as a shared infrastructure problem, not a personal compliance decision.