CASRAI Dictionary

Tag: responsible use of metrics

  • Responsible Use of Metrics: Comparing UK University DORA Guidance

    The responsible use of metrics means applying quantitative research indicators — citation counts, field-weighted citation impact, grant income — only to inform and support expert peer judgement, never to replace it, in line with the San Francisco Declaration on Research Assessment (DORA). Cambridge, Exeter, Edinburgh and UCD have each published DORA-aligned guidance for their own institutions, but a side-by-side reading shows the four documents converge on principle and diverge sharply on governance, prescriptiveness and review discipline.

    Responsible research metrics is the umbrella term for institutional policies that constrain how bibliometric and altmetric indicators may be used in hiring, promotion and funding decisions, so that no single number is treated as a proxy for research quality.

    What does “responsible use of metrics” actually require?

    DORA’s own guidance on the responsible use of quantitative indicators sets out five criteria that any institutional policy should meet: metrics use should be clear, be transparent, be specific, be contextual, and be fair, according to sfdora.org’s published guidance document. Separately, the UK’s Forum for Responsible Research Metrics — convened by Universities UK following the 2015 Metric Tide report — frames the same territory as five R’s: robustness, humility, transparency, diversity and reflexivity.

    Every institutional statement reviewed here traces back to the same three source documents: DORA (2012), the Leiden Manifesto for Research Metrics (2015), and the Metric Tide (2015, updated as Harnessing the Metrics Tide in 2022). What differs is how each university translates those shared principles into binding local policy — and that is where the real variation, and the real risk of inconsistent practice, sits.

    Where Cambridge, Exeter, Edinburgh and UCD guidance converges

    All four institutions state unambiguously that quantitative metrics must support, not supplant, qualitative expert assessment. All four are DORA signatories and all four explicitly rule out using the Journal Impact Factor as a proxy for the quality of an individual output or researcher.

    • Metrics must be applied at the correct level of granularity — never using a journal-level or institution-level number to judge an individual.
    • Comparisons between individuals must account for career stage, career breaks and part-time working.
    • Any metric used in assessment must be disclosed in advance to the people being assessed.
    • Metrics and their underlying datasets must be periodically reviewed for continued fitness of purpose.

    Edinburgh and Exeter are also both signatories to the Coalition for Advancing Research Assessment (CoARA), which each joined in 2022, committing them additionally to phasing out inappropriate use of the h-index alongside the Journal Impact Factor.

    Where the four institutions’ guidance diverges

    Beneath the shared principles, the four documents take genuinely different institutional forms — a distinction that matters more than the principles themselves for anyone trying to replicate or benchmark a policy.

    Institution Format Governance body Distinguishing feature
    University of Cambridge High-level institutional principles, devolved Individual Schools and Faculties No single university-wide rulebook; Schools write discipline-specific DORA implementation policies
    University of Exeter Nine enumerated principles (published April 2022) Responsible Metrics Champions Group, plus a DORA Champions network in Colleges and Services Explicitly built on the UCL Principles for the responsible use of bibliometrics as its starting template
    University of Edinburgh Five numbered institutional commitments (“The University will…”) Research Policy Group (2019 approval); Research Strategy Group (2025 re-approval) Only one of the four with a published review cadence — first approved April 2019, reviewed May 2025, next review Spring 2028
    University College Dublin Single institutional statement synthesising three peer frameworks Working Group on the Responsible Use of Research Metrics, reporting to the Research, Innovation and Impact Group (RIIG) Most explicit on equality, diversity and inclusion factors — names career breaks, statutory leave and part-time working directly in the policy text

    Exeter’s document is the most technically granular of the four, naming specific indicator products — Field-Weighted Citation Impact from Scopus/SciVal, Field Citation Ratio and Relative Citation Ratio from Digital Science, Category Normalised Citation Impact from Web of Science — and warning explicitly against mixing metrics from different bibliometric providers within the same assessment exercise. Edinburgh is the most procedurally binding, with a stated review cycle and a named committee for re-approval. Cambridge is the most devolved, deliberately declining to impose a single university-wide metric policy in favour of discipline-appropriate local rules. UCD is the most EDI-forward, embedding equity language directly into its core commitments rather than treating it as a supporting principle.

    What gaps remain for institutions without a dedicated policy?

    Cambridge, Exeter, Edinburgh and UCD each have a named committee, a published document and — in Edinburgh’s case — a fixed review date. Many smaller and teaching-intensive institutions have none of this. Several UK universities that rank prominently for “responsible use of metrics” searches — including library subject guides from institutions such as Derby, Plymouth and Sunderland — publish summaries of DORA’s principles rather than institutionally approved governance statements.

    That distinction is not cosmetic. A library guide can explain what responsible metrics are; it cannot bind a promotion committee the way a document approved by a Research Policy Group, a Champions Group or an RIIG can. Institutions without a dedicated policy and a named approving body carry a structural gap: staff have no enforceable assurance that a hiring panel or REF preparation exercise will actually follow the principles a library page describes. For research administrators at smaller institutions, the practical route is not to draft new principles from scratch but to adapt an existing framework — Exeter’s document explicitly credits the UCL Principles as its own starting point, and UCD’s statement was built after reviewing three existing peer institutions’ policies, showing that adaptation, not original drafting, is the established norm.

    Answer-first Q&A on research metrics

    What are the four types of metrics used in research assessment?

    Institutional guidance, including Exeter’s, groups research indicators into institutional or discipline-level indicators (rankings, field-weighted citation impact), output-level indicators (citation counts, Journal Impact Factor, altmetrics), research activity indicators (grant income, PGR numbers) and individual-focused indicators (h-index, highly-cited rankings) — each requiring different safeguards against misuse.

    What is the use of metrics in responsible research assessment?

    Metrics provide contextual, supporting evidence alongside qualitative peer review — never a standalone verdict. Under DORA and the institutional statements reviewed here, quantitative indicators may inform hiring, promotion and funding decisions only when disclosed in advance, appropriately normalised and applied at the correct level of granularity.

    What are examples of responsible research metrics?

    Commonly cited examples include field-weighted citation impact, altmetrics, grant income and postgraduate research supervision counts, used as part of a discipline-appropriate “basket of measures” rather than in isolation. Journal Impact Factor and raw h-index are explicitly excluded as individual-level proxies by every institution examined here.

    Implications for research administrators

    For research administrators, the comparison points to a practical hierarchy of maturity: a published statement with no named governance body (the entry point most smaller institutions can reach quickly); a statement with a standing committee (Exeter’s Champions Group, UCD’s Working Group reporting into RIIG); and a statement with a fixed, published review cycle (Edinburgh’s model, next due Spring 2028). Institutions preparing for REF2029 have a direct incentive to close this gap now, since metrics played a limited but real role in informing peer review for REF2021 and several universities’ Codes of Practice explicitly reserve the right to expand that role.

    The direction of travel across the sector is unambiguous: DORA and CoARA signatory numbers continue to grow, and the Forum for Responsible Research Metrics gives every institution — large or small — a ready-made template rather than a blank page. The remaining work is not persuasion but implementation: naming a governance body, setting a review date, and publishing the document where staff undergoing assessment can actually find it.

  • Forum for Responsible Research Metrics Explained

    The Forum for Responsible Research Metrics is the UK’s national coordinating body for the responsible use of research metrics. Established in 2016 following the 2015 Metric Tide review, it advises UK higher education funding bodies on metrics use in the Research Excellence Framework (REF) and promotes alignment with DORA and CoARA principles.

    The Forum for Responsible Research Metrics is an independent, sector-wide group of UK research funders, sector bodies and infrastructure experts convened to promote fair and transparent use of quantitative indicators in research assessment. It sits alongside — but is distinct from — the global DORA declaration and the European CoARA coalition, and its guidance shapes how UK institutions design metrics policy ahead of the REF.

    What is the Forum for Responsible Research Metrics?

    The Forum for Responsible Research Metrics is a UK sector body, not a regulator. It has no statutory powers and cannot compel institutions to adopt any metric or policy. Instead it functions as an advisory and convening body, bringing together funders, universities and data-infrastructure providers to agree shared principles for using metrics responsibly in research assessment.

    Its core functions, as set out at its founding, are threefold: advise the UK higher education funding bodies on metrics use in the REF; provide advocacy and leadership on responsible metrics within the UK sector; and establish links with equivalent international initiatives. The Forum is administratively supported by Universities UK (UUK), which convenes its meetings and publishes its outputs.

    When and why was the Forum established?

    The Forum’s origins trace directly to a government-commissioned review. In 2014 the then Higher Education Funding Council for England (HEFCE) — now folded into UKRI’s Research England — commissioned an independent expert group to examine the use of metrics in research evaluation, particularly within the REF. The resulting report, The Metric Tide, was published on 9 July 2015.

    The Metric Tide concluded that metrics could be a useful adjunct to peer review but warned against their use as a substitute for expert judgement, and recommended that institutions consider signing DORA or applying its principles. It also recommended that a UK-wide body be established to advise funders on metrics use in the REF, provide sector leadership, and build international links. The Forum for Responsible Research Metrics was convened in 2016 to fulfil that recommendation.

    • 2014 — HEFCE commissions the independent metrics review
    • 9 July 2015 — The Metric Tide report is published
    • 2016 — The Forum for Responsible Research Metrics is convened, supported by Universities UK
    • 2021 — Forum advice informs metrics use across the three REF2021 assessment elements
    • December 2022 — Sector commentary (Wonkhe) calls for an expanded Forum remit, including holding data providers to account

    How does the Forum relate to DORA and CoARA?

    The Forum, DORA and CoARA are three distinct bodies with overlapping but separate mandates, and conflating them is a common source of confusion for research offices drafting policy. The table below sets out how each operates and how they connect to one another.

    Body Founded Scope Core mechanism Link to the Forum
    Forum for Responsible Research Metrics 2016 UK-wide, advisory to HE funding bodies Advises funders on metrics use in the REF; monitors DORA/CoARA uptake —
    DORA (San Francisco Declaration on Research Assessment) 2013 International, institution/publisher signatory declaration Signatories pledge not to use the Journal Impact Factor as a proxy for individual quality in hiring, promotion or funding decisions The Metric Tide recommended UK institutions sign DORA; the Forum promotes and tracks its adoption
    CoARA (Coalition for Advancing Research Assessment) 2022 Pan-European coalition, member commitments Signatories commit to reforming assessment criteria and procedures over a defined implementation period Complementary European framework; several UK signatories hold both DORA and CoARA membership alongside Forum-aligned institutional policy
    REF (Research Excellence Framework) 2014 (successor to the RAE) UK-wide, all HE institutions Periodic peer-review-led assessment of research quality, impact and environment The Forum’s core client — its advice shapes how funding bodies use metrics within REF criteria

    In practice, the Forum does not ask institutions to sign anything. DORA and CoARA are commitments an institution opts into; the Forum’s guidance is advisory input into how UK funders design and apply metrics within a statutory national exercise.

    What is the Forum’s role in the REF?

    The Forum’s most concrete, recurring function is advising UK higher education funding bodies on how metrics should — and should not — be used within the REF’s three assessment elements: outputs, impact and environment. This advice fed directly into REF2021 guidance and is expected to inform preparation for the next exercise, REF 2029.

    UKRI’s Research England states that the Forum works to improve the data infrastructure underpinning metric use and the broader culture around research metrics, not just the rules for a single assessment cycle. That distinction matters for research offices: Forum guidance is a standing reference point for metrics governance, not a one-off REF submission checklist.

    • Outputs — metrics may inform but must not substitute for peer review of research quality
    • Impact — quantitative indicators supplement, rather than replace, narrative impact case studies
    • Environment — metrics contribute contextual evidence on research culture and infrastructure

    What does the Forum’s guidance mean for research offices?

    For research administrators building or reviewing a metrics policy, Forum guidance sets a de facto national baseline: metrics should be used transparently, contextually, and never as an automatic proxy for quality in hiring, promotion or funding decisions. This mirrors DORA’s core ask but is framed specifically for REF-facing institutional practice.

    Institutional research offices typically apply this in three ways: auditing existing use of journal-level and author-level metrics in internal review processes; documenting which indicators are used for which decisions and why; and aligning local policy statements with DORA and, where relevant, CoARA commitments so REF-facing metrics governance is not developed in isolation. Institutions building or auditing research assessment policy can find related structural guidance in CASRAI’s research administration resources.

    The December 2022 sector call to expand the Forum’s remit — including holding metrics data providers to account — signals that its scope is likely to widen beyond REF-facing advice toward broader accountability for the commercial infrastructure that supplies citation and impact data. Research offices should treat current Forum guidance as a floor, not a ceiling, when drafting policy ahead of REF 2029.

    Frequently asked questions

    Who chairs the UK Forum for Responsible Research Metrics?

    As of Universities UK’s most recently published listing, the Forum is chaired by Professor Max Lu, Vice-Chancellor of the University of Surrey. The Forum itself comprises representatives from UK research funders, sector bodies and data-infrastructure organisations, convened administratively by Universities UK.

    Is the Forum for Responsible Research Metrics the same as DORA?

    No. DORA is a global signatory declaration institutions and individuals opt into; the Forum is a UK sector body advising funders on metrics within the REF. The Metric Tide review recommended UK institutions sign DORA, and the Forum monitors and promotes that uptake without being DORA itself.

    What did the Metric Tide report recommend?

    Published 9 July 2015, The Metric Tide recommended that metrics support but never replace peer review, that institutions consider signing DORA, and that a national Forum be established to advise funders on responsible metrics use in the REF and build international links.

    Does the Forum’s advice apply to REF 2029?

    Yes. The Forum’s advisory role is standing, not exercise-specific, and its guidance on metrics in outputs, impact and environment assessment is expected to inform funding-body preparation for REF 2029 as it did for REF2021.

    What’s next for responsible metrics in the UK?

    With REF 2029 preparation underway and sector pressure to widen the Forum’s remit toward data-provider accountability, UK research offices should expect Forum guidance to evolve rather than stay fixed. Institutions that align internal metrics policy with Forum principles now, rather than at the point of REF submission, will face less rework as that remit expands.