CASRAI Dictionary

Tag: guidance on the responsible use of metrics

  • Quantitative Indicators in Research Assessment: A Hiring and Promotion Panel Guide

    Under DORA and the CoARA Agreement, quantitative indicators such as the Journal Impact Factor and h-index must never substitute for expert peer judgement in hiring and promotion decisions — they may only inform it, applied with clarity, transparency, specificity, context and fairness, alongside a broader account of a candidate’s contributions.

    Quantitative indicators in research assessment are numerical proxies for research activity — citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetrics — used, under explicit caveats, to inform rather than replace qualitative evaluation of a researcher’s work.

    Research offices translating this principle into a hiring or promotion brief face a harder question than “which metrics are banned?” Panels need operational wording for the call, the assessor briefing and the case file. This guide sets out what DORA, the CoARA Agreement and the UK’s Forum for Responsible Research Metrics concretely require, and how to turn that into panel-ready criteria.

    Contents

    What counts as a quantitative indicator in research assessment?

    A quantitative indicator is any numerical measure derived from research outputs or activity: citation counts, the h-index, the Journal Impact Factor (JIF), field-normalised citation ratios, grant income, patent counts and altmetric mentions all qualify. None was designed to certify the quality of a single article or a single person’s contribution.

    The University of York’s policy for research evaluation using quantitative data, approved by its Research Committee in November 2017, makes the distinction explicit: indicators are informative at departmental or institutional level, but “the assessment of individual research performance using solely quantitative indicators is not supported.” That collective-versus-individual distinction is the fault line every hiring and promotion policy has to draw.

    What does DORA require for hiring and promotion panels?

    The San Francisco Declaration on Research Assessment (DORA), agreed in December 2012, states a single unambiguous prohibition that panels must apply: do not use journal-based metrics, such as the Journal Impact Factor, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions. That sentence, not a general suspicion of numbers, is DORA’s operative rule for panels.

    DORA does not ban quantitative indicators outright. Its 2024 guidance document on the responsible use of quantitative indicators, produced by a DORA task force chaired by Professor Stephen Curry and published via Zenodo, sets out five principles that must govern any indicator a panel does choose to use: be clear, be transparent, be specific, be contextual, and be fair. These are DORA’s own words. Some AI-generated summaries currently paraphrase this as “the five Cs” — clarity, context, calibration, care, credit — a mnemonic that does not appear anywhere in DORA’s published guidance; panels drafting criteria should cite DORA’s actual five principles instead.

    Applied to a panel: state which indicator is being consulted and why (clear); disclose the data source and calculation method (transparent); tie the indicator to the specific claim it supports, not a general quality judgement (specific); benchmark against discipline and career stage (contextual); and check for bias against gender, geography, career breaks or non-traditional outputs (fair).

    What does the CoARA Agreement commit panels to?

    The Coalition for Advancing Research Assessment (CoARA) launched its Agreement on Reforming Research Assessment in 2022, since signed by several hundred universities, funders, national agencies and learned societies across Europe and beyond. The Agreement sets ten commitments; its core, non-negotiable commitment is to “abandon inappropriate uses in research assessment of journal- and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index.”

    Beyond that prohibition, CoARA’s commitments push panels toward qualitative peer review as the primary method, recognition of a wider range of outputs — datasets, software, protocols, policy engagement, mentoring and open-science practice — and narrative formats such as narrative CVs that let candidates describe contributions in their own words.

    The table below compares the three frameworks a UK or European research office is most likely to be asked to reconcile.

    Framework Origin and scope Core requirement for hiring/promotion Status of quantitative indicators
    DORA Global; agreed San Francisco, December 2012 Do not use JIF as a surrogate for individual quality in hiring, promotion or funding decisions Conditional use only, governed by five principles: clear, transparent, specific, contextual, fair
    CoARA Agreement Pan-European coalition; launched 2022 Core commitment to abandon inappropriate JIF/h-index use in individual assessment Indicators permitted only to support, not replace, qualitative peer review
    Forum for Responsible Research Metrics (UK) UK sector body, stemming from The Metric Tide (Wilsdon et al., HEFCE, 2015) Institutions asked to publish a responsible-metrics statement covering hiring/promotion criteria Five dimensions: robustness, humility, transparency, diversity, reflexivity

    Translating principles into concrete panel criteria

    Principles do not write themselves into a job description. A defensible panel criteria set, translating DORA, CoARA and Forum for Responsible Research Metrics guidance into working practice, includes:

    • State in the call and case-file template that the Journal Impact Factor, h-index and journal rank will not be used as proxies for individual quality (DORA’s core recommendation).
    • Offer or require a narrative CV alongside, or instead of, a conventional publication list, so data, software, mentoring and open-science contributions are visible to assessors.
    • If citation data is used at all, require field-normalised indicators rather than raw counts, and disclose the source database in the case file.
    • Credit non-publication outputs explicitly in the assessment rubric, consistent with CoARA’s broadened-recognition commitments.
    • Brief panel members on indicator limitations before each cycle, per the Forum for Responsible Research Metrics’ “humility” dimension.
    • Record, for each case, which indicators (if any) were consulted and the specific claim they supported (DORA’s “transparent” and “specific” principles).
    • Review the criteria annually, reflecting the “reflexivity” dimension shared by the Leiden Manifesto (Hicks et al., Nature, 2015) and the Forum for Responsible Research Metrics.

    A useful complementary vocabulary for the “credit non-publication outputs” step is a structured contributor-role taxonomy. CASRAI originated the CRediT contributor role taxonomy in 2014; the standard is now stewarded by NISO as ANSI/NISO Z39.104-2022. Panels reviewing narrative CVs can use CRediT’s fourteen roles to make specific, verifiable contribution claims — distinguishing data curation from formal analysis, for example — rather than relying on author order or citation counts as a proxy for who did what.

    Frequently asked questions

    What are quantitative indicators in research assessment?

    Quantitative indicators are numerical measures of research activity, including citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetric mentions. DORA’s guidance treats them as descriptive data points requiring context, not standalone quality scores, and warns against using any single indicator in isolation.

    Does DORA allow any use of quantitative indicators in hiring and promotion?

    Yes, conditionally. DORA does not ban indicators outright; it prohibits journal-based metrics like the Journal Impact Factor as a surrogate for individual quality in hiring, promotion or funding decisions. Where indicators are used, DORA’s five principles — clear, transparent, specific, contextual, fair — must govern their application.

    What does the CoARA Agreement require of hiring and promotion panels?

    CoARA’s core commitment obliges signatories to abandon inappropriate use of journal- and publication-based metrics, particularly the Journal Impact Factor and h-index, in individual assessment. Panels must prioritise qualitative peer judgement, broaden recognised output types, and adopt formats such as narrative CVs.

    What is a narrative CV, and is it required under responsible metrics guidance?

    A narrative CV lets candidates describe significant contributions — including data, software, mentoring and open-science practice — in their own words, rather than through a publication-and-citation list. DORA and CoARA both recommend narrative formats to support qualitative review, though neither makes them a formal, binding requirement.

    Implications for research offices

    UK institutions face a specific reconciliation problem: government is considering bibliometric data as an optional component of the next Research Excellence Framework exercise, REF 2029, at discipline and institutional level, even as DORA and CoARA prohibit citation-based proxies at the level of the individual hire. Policy wording needs to keep these two scales distinct — permitting aggregate bibliometric reporting upward to funders while barring the same data from an individual case file.

    The direction of travel across DORA and CoARA signatories is consistent: fewer single-number thresholds, more disclosed and contextualised indicator use, and a growing expectation that panels can explain, in writing, which evidence supported which judgement. Research offices that build this documentation habit now, rather than waiting for a funder or auditor to ask, will find each subsequent cycle easier to defend, not harder.

  • Journal Impact Factor Reform After DORA, CoARA: What the Evidence Shows

    Journal impact factor reform is the shift, led by the San Francisco Declaration on Research Assessment (DORA) and the Coalition for Advancing Research Assessment (CoARA), away from journal-level metrics as proxies for individual researcher quality in hiring, promotion and tenure. Published institutional policy audits show the shift is real but partial: peer-reviewed analysis of promotion documents finds journal impact factor (JIF) language persists at a large minority of research-intensive universities even where DORA has been signed.

    DORA is a 2012 declaration, now signed by more than 25,000 individuals and organisations worldwide, that asks institutions to stop using the JIF as a substitute measure of the quality of individual research articles in funding, appointment and promotion decisions. CoARA, launched in 2022 under the Agreement on Reforming Research Assessment (ARRA), extends the same principle into a formal European coalition with published implementation commitments and a 2029 systemic-reform deadline for signatory organisations.

    What DORA and CoARA Actually Require

    DORA and CoARA are not enforcement bodies; they are voluntary commitments that institutions self-implement. Neither has an audit or sanction mechanism of its own, which is precisely why independent policy audits — not signatory counts — are the only reliable evidence of actual practice change.

    Feature DORA CoARA
    Launched 2012 (San Francisco) 2022, under the ARRA agreement
    Core mechanism Voluntary declaration; no journal-level metrics as a proxy for article or researcher quality Ten formal commitments plus a published multi-year action plan per signatory
    Geographic base Global Predominantly European, growing globally
    Compliance check None — self-reported case studies only National chapters and working groups; EC-commissioned baseline studies
    Reported uptake 25,000+ individual and organisational signatories Around 450 European higher-education institutions, roughly 13% of the sector, per the CoARA-by-numbers uptake study

    Does Signing DORA or CoARA Actually Change Hiring Practice?

    The most direct published audit remains McKiernan et al.’s 2019 eLife meta-research study, which coded the review, promotion and tenure (RPT) documents of 129 US and Canadian universities. The findings are the clearest evidence-based answer available on this question.

    • The JIF was mentioned in RPT documents at 23% of all sampled institutions, rising to 40% among research-intensive (R-type) universities.
    • Of institutions that mentioned the JIF, 87% used it in a supportive context — encouraging its use — and none explicitly prohibited it, even where the parent university had signed DORA.
    • Where JIF appeared, 63% of mentions linked it to “quality”, 40% to impact or significance, and 20% to prestige or reputation — the exact conflation DORA was written to dismantle.

    A follow-up 2024 DORA-commissioned qualitative study of US faculty hiring and tenure assessments reached a similar conclusion: departmental practice frequently lags institutional signature, because RPT criteria are set and applied at department or faculty level, not centrally by the office that signed the declaration. This decentralisation gap is the single most consistent finding across the audit literature and is the primary reason blanket “signed DORA” claims cannot be read as evidence of changed practice.

    Which Institutions Have Implemented Reform, and How

    Where reform has taken hold, it has required a specific policy rewrite, not just a signature. Three documented examples illustrate the range of implementation depth.

    • Utrecht University (Netherlands) signed DORA in 2019 and, by 2022, had formally removed the JIF from all hiring and promotion criteria university-wide as part of its Recognition and Rewards programme, replacing it with team-science and open-science indicators — reported by Nature in July 2021.
    • University of Calgary revised its GFC Academic Staff Criteria and Processes Handbook after signing DORA, explicitly incorporating DORA’s principles into the formal criteria used in tenure and promotion committees, according to the university’s own published case study.
    • CoARA national chapters, including Spain’s, have published sector-wide mapping reports tracking which member institutions have moved from commitment to documented policy change, rather than relying on signature counts alone.

    At the coalition level, the European Commission’s independently commissioned ARRA baseline study — published in mid-2026 — found that engagement is broad and growing but that implementation progress across signatory organisations is uneven and constrained by limited institutional capacity, particularly outside research-intensive universities and outside Western Europe.

    Why Reform Stalls: Barriers Identified in the Audits

    The audit literature converges on a consistent set of structural barriers, distinct from a simple lack of institutional will.

    • Decentralised authority. Central research offices sign declarations; departments and faculties write and apply RPT criteria, creating an implementation gap that persists for years.
    • No enforceable alternative metric. DORA and CoARA prescribe what institutions should stop doing more clearly than what should replace it, leaving evaluators to fall back on familiar journal-based shortcuts under time pressure.
    • Reviewer and panel habit. External referees and appointment panels — often from non-signatory institutions — continue to reference journal prestige informally, even where the host institution’s written policy is silent on the JIF.
    • Absence of external audit. Because neither DORA nor CoARA verifies signatory compliance, self-reported case studies dominate the evidence base, which is why the McKiernan-style document audit remains the field’s methodological benchmark.

    Common Questions on DORA, CoARA and Impact Factor Reform

    Has signing DORA actually changed hiring practice at universities?

    Partially. Published audits of promotion and tenure documents show the journal impact factor still appears at roughly a quarter to two-fifths of institutions, including many DORA signatories, because departmental criteria often lag the institutional-level declaration by years.

    What is the difference between DORA and CoARA?

    DORA is a global, individually signed declaration with no formal implementation mechanism, launched in 2012. CoARA is a European-centred coalition with ten binding commitments and a published multi-year action plan per signatory institution, launched in 2022.

    Do DORA-signatory universities still use the impact factor in tenure decisions?

    Some do. A 2019 eLife audit found 87% of institutions that mentioned the JIF in tenure documents used it in a supportive, encouraging context, regardless of DORA status, showing that signature alone does not remove journal metrics from evaluation practice.

    What replaces the impact factor under responsible research assessment reform?

    Reformed institutions typically adopt narrative CVs, qualitative peer review, and broadened output categories — datasets, software, mentorship, open-science contributions — rather than a single quantitative substitute metric, per CoARA’s core commitments.

    Implications for Research Administrators and Institutional Leaders

    For research administrators, the audit evidence carries a practical conclusion: a DORA or CoARA signature is a governance commitment, not a completed policy change. Verifying reform therefore requires the same document-level audit McKiernan’s team used — checking actual RPT, hiring and promotion wording for JIF language — rather than relying on signatory-list membership as a proxy for compliance.

    Institutions serious about research assessment governance should treat departmental RPT criteria, external referee guidance and appointment panel training as the three concrete levers that determine whether a declaration changes behaviour. Coalition membership sets direction; department-level document rewrites are what the evidence shows actually moves practice.

    The trajectory across both audits and coalition-level reporting points toward continued, uneven reform rather than wholesale abandonment of journal metrics. CoARA’s 2029 milestone and the growing base of institutional case studies mean the evidence base for measuring real change — as opposed to signed intent — will keep expanding over the next several years.

  • Responsible Use of Metrics: Comparing UK University DORA Guidance

    The responsible use of metrics means applying quantitative research indicators — citation counts, field-weighted citation impact, grant income — only to inform and support expert peer judgement, never to replace it, in line with the San Francisco Declaration on Research Assessment (DORA). Cambridge, Exeter, Edinburgh and UCD have each published DORA-aligned guidance for their own institutions, but a side-by-side reading shows the four documents converge on principle and diverge sharply on governance, prescriptiveness and review discipline.

    Responsible research metrics is the umbrella term for institutional policies that constrain how bibliometric and altmetric indicators may be used in hiring, promotion and funding decisions, so that no single number is treated as a proxy for research quality.

    What does “responsible use of metrics” actually require?

    DORA’s own guidance on the responsible use of quantitative indicators sets out five criteria that any institutional policy should meet: metrics use should be clear, be transparent, be specific, be contextual, and be fair, according to sfdora.org’s published guidance document. Separately, the UK’s Forum for Responsible Research Metrics — convened by Universities UK following the 2015 Metric Tide report — frames the same territory as five R’s: robustness, humility, transparency, diversity and reflexivity.

    Every institutional statement reviewed here traces back to the same three source documents: DORA (2012), the Leiden Manifesto for Research Metrics (2015), and the Metric Tide (2015, updated as Harnessing the Metrics Tide in 2022). What differs is how each university translates those shared principles into binding local policy — and that is where the real variation, and the real risk of inconsistent practice, sits.

    Where Cambridge, Exeter, Edinburgh and UCD guidance converges

    All four institutions state unambiguously that quantitative metrics must support, not supplant, qualitative expert assessment. All four are DORA signatories and all four explicitly rule out using the Journal Impact Factor as a proxy for the quality of an individual output or researcher.

    • Metrics must be applied at the correct level of granularity — never using a journal-level or institution-level number to judge an individual.
    • Comparisons between individuals must account for career stage, career breaks and part-time working.
    • Any metric used in assessment must be disclosed in advance to the people being assessed.
    • Metrics and their underlying datasets must be periodically reviewed for continued fitness of purpose.

    Edinburgh and Exeter are also both signatories to the Coalition for Advancing Research Assessment (CoARA), which each joined in 2022, committing them additionally to phasing out inappropriate use of the h-index alongside the Journal Impact Factor.

    Where the four institutions’ guidance diverges

    Beneath the shared principles, the four documents take genuinely different institutional forms — a distinction that matters more than the principles themselves for anyone trying to replicate or benchmark a policy.

    Institution Format Governance body Distinguishing feature
    University of Cambridge High-level institutional principles, devolved Individual Schools and Faculties No single university-wide rulebook; Schools write discipline-specific DORA implementation policies
    University of Exeter Nine enumerated principles (published April 2022) Responsible Metrics Champions Group, plus a DORA Champions network in Colleges and Services Explicitly built on the UCL Principles for the responsible use of bibliometrics as its starting template
    University of Edinburgh Five numbered institutional commitments (“The University will…”) Research Policy Group (2019 approval); Research Strategy Group (2025 re-approval) Only one of the four with a published review cadence — first approved April 2019, reviewed May 2025, next review Spring 2028
    University College Dublin Single institutional statement synthesising three peer frameworks Working Group on the Responsible Use of Research Metrics, reporting to the Research, Innovation and Impact Group (RIIG) Most explicit on equality, diversity and inclusion factors — names career breaks, statutory leave and part-time working directly in the policy text

    Exeter’s document is the most technically granular of the four, naming specific indicator products — Field-Weighted Citation Impact from Scopus/SciVal, Field Citation Ratio and Relative Citation Ratio from Digital Science, Category Normalised Citation Impact from Web of Science — and warning explicitly against mixing metrics from different bibliometric providers within the same assessment exercise. Edinburgh is the most procedurally binding, with a stated review cycle and a named committee for re-approval. Cambridge is the most devolved, deliberately declining to impose a single university-wide metric policy in favour of discipline-appropriate local rules. UCD is the most EDI-forward, embedding equity language directly into its core commitments rather than treating it as a supporting principle.

    What gaps remain for institutions without a dedicated policy?

    Cambridge, Exeter, Edinburgh and UCD each have a named committee, a published document and — in Edinburgh’s case — a fixed review date. Many smaller and teaching-intensive institutions have none of this. Several UK universities that rank prominently for “responsible use of metrics” searches — including library subject guides from institutions such as Derby, Plymouth and Sunderland — publish summaries of DORA’s principles rather than institutionally approved governance statements.

    That distinction is not cosmetic. A library guide can explain what responsible metrics are; it cannot bind a promotion committee the way a document approved by a Research Policy Group, a Champions Group or an RIIG can. Institutions without a dedicated policy and a named approving body carry a structural gap: staff have no enforceable assurance that a hiring panel or REF preparation exercise will actually follow the principles a library page describes. For research administrators at smaller institutions, the practical route is not to draft new principles from scratch but to adapt an existing framework — Exeter’s document explicitly credits the UCL Principles as its own starting point, and UCD’s statement was built after reviewing three existing peer institutions’ policies, showing that adaptation, not original drafting, is the established norm.

    Answer-first Q&A on research metrics

    What are the four types of metrics used in research assessment?

    Institutional guidance, including Exeter’s, groups research indicators into institutional or discipline-level indicators (rankings, field-weighted citation impact), output-level indicators (citation counts, Journal Impact Factor, altmetrics), research activity indicators (grant income, PGR numbers) and individual-focused indicators (h-index, highly-cited rankings) — each requiring different safeguards against misuse.

    What is the use of metrics in responsible research assessment?

    Metrics provide contextual, supporting evidence alongside qualitative peer review — never a standalone verdict. Under DORA and the institutional statements reviewed here, quantitative indicators may inform hiring, promotion and funding decisions only when disclosed in advance, appropriately normalised and applied at the correct level of granularity.

    What are examples of responsible research metrics?

    Commonly cited examples include field-weighted citation impact, altmetrics, grant income and postgraduate research supervision counts, used as part of a discipline-appropriate “basket of measures” rather than in isolation. Journal Impact Factor and raw h-index are explicitly excluded as individual-level proxies by every institution examined here.

    Implications for research administrators

    For research administrators, the comparison points to a practical hierarchy of maturity: a published statement with no named governance body (the entry point most smaller institutions can reach quickly); a statement with a standing committee (Exeter’s Champions Group, UCD’s Working Group reporting into RIIG); and a statement with a fixed, published review cycle (Edinburgh’s model, next due Spring 2028). Institutions preparing for REF2029 have a direct incentive to close this gap now, since metrics played a limited but real role in informing peer review for REF2021 and several universities’ Codes of Practice explicitly reserve the right to expand that role.

    The direction of travel across the sector is unambiguous: DORA and CoARA signatory numbers continue to grow, and the Forum for Responsible Research Metrics gives every institution — large or small — a ready-made template rather than a blank page. The remaining work is not persuasion but implementation: naming a governance body, setting a review date, and publishing the document where staff undergoing assessment can actually find it.