CASRAI Dictionary

Tag: forum for responsible research metrics

  • Leiden Manifesto Checklist for Research Offices

    The Leiden Manifesto for Research Metrics sets out ten principles, published as a comment in Nature in 2015, for the responsible use of quantitative indicators in research evaluation. Research offices can convert each principle into a direct audit question, testing whether KPI dashboards, promotion criteria and grant-review rubrics rely on a single metric, ignore field norms, or substitute for qualitative judgement.

    The Leiden Manifesto for Research Metrics is a ten-principle framework for the responsible use of bibliometric and other quantitative indicators in evaluating research, published by Diana Hicks, Paul Wouters, Ludo Waltman, Sarah de Rijcke and Ismael Rafols in Nature on 22 April 2015. It was formulated at the 19th International Conference on Science and Technology Indicators, held in Leiden, the Netherlands, in September 2014, and has since been cited more than 4,000 times, according to Google Scholar’s tracking of the original paper.

    What is the Leiden Manifesto for Research Metrics?

    The Leiden Manifesto is a response to what its authors called “impact-factor obsession” — the tendency of universities, funders and promotion committees to substitute a single number for expert judgement. It does not ban metrics. It requires that quantitative indicators support, rather than replace, informed peer assessment of research quality.

    The manifesto’s home institution is the Centre for Science and Technology Studies (CWTS) at Leiden University, where co-author Paul Wouters served as director. CWTS also produces the CWTS Leiden Ranking, a separate bibliometrics-based university ranking — a distinction research offices should not conflate when citing the source.

    What are the ten principles of the Leiden Manifesto?

    Each principle addresses a specific failure mode observed in metric-driven research assessment. The table below states each principle exactly as published, alongside the practical audit question a research office should ask of its own KPI or promotion framework.

    # Principle (Hicks et al., 2015) Audit question for your office
    1 Quantitative evaluation should support qualitative, expert assessment Does any committee decision rest on a metric alone, with no narrative peer input?
    2 Measure performance against the research missions of the institution, group or researcher Are KPIs generic, or tailored to the unit’s stated mission (teaching-intensive, applied, translational)?
    3 Protect excellence in locally relevant research Does the framework penalise work published in non-English or regionally focused outlets?
    4 Keep data collection and analytical processes open, transparent and simple Can an academic reproduce their own score from publicly documented methodology?
    5 Allow those evaluated to verify data and analysis Is there a formal, timely route to challenge or correct metric data before a decision is made?
    6 Account for variation by field in publication and citation practices Are raw citation counts compared across disciplines without field normalisation?
    7 Base assessment of individual researchers on a qualitative judgement of their portfolio Does promotion criteria require a portfolio narrative, or just an h-index threshold?
    8 Avoid misplaced concreteness and false precision Are decimal-point differences in impact factor or citation rate treated as meaningful?
    9 Recognise the systemic effects of assessment and indicators Has the office assessed whether its KPIs create incentives to game submission counts or venues?
    10 Scrutinise indicators regularly and update them Is there a scheduled review cycle for the KPI framework itself, not just for scores against it?

    How can a research office audit its KPI and promotion framework against it?

    Running the manifesto as a live audit tool means working through each principle against real artefacts: the appraisal form, the promotion rubric, and the departmental dashboard.

    1. Mark every clause in the promotion/tenure criteria naming a specific metric (impact factor, h-index, citation count).
    2. Check each marked clause has a qualitative narrative requirement alongside it (Principles 1 and 7).
    3. Confirm KPI targets are set per unit mission, not copied institution-wide (Principle 2).
    4. Check non-English-language or applied outputs score on the same scale as high-impact-journal outputs (Principle 3).
    5. Verify each dashboard metric’s data source and calculation method is documented and accessible (Principles 4 and 5).
    6. Confirm citation indicators are field-normalised, not raw counts compared across disciplines (Principle 6).
    7. Look for false precision — ranking staff by two-decimal citation averages (Principle 8).
    8. Ask whether the KPI framework has driven any unintended behaviour, such as salami-slicing publications or discouraging risky research (Principle 9).
    9. Set a fixed review date for the framework itself, independent of individual appraisal cycles (Principle 10).

    A framework that fails more than two or three of these checks is not aligned with the manifesto, regardless of how sophisticated its dashboard software looks. The most common failure in practice is Principle 6: comparing raw citation counts across a mathematics department and a cell biology department, where top-ranked mathematics journals carry impact factors around 3 while top-ranked cell biology journals carry impact factors around 30 — a field-scale gap the manifesto’s authors cite directly as evidence that uncorrected cross-field comparison is meaningless.

    How does the Leiden Manifesto compare with DORA and CoARA?

    The Leiden Manifesto did not appear in isolation. The 2013 San Francisco Declaration on Research Assessment (DORA) preceded it, while the Coalition for Advancing Research Assessment (CoARA) has since built a sector-wide agreement on reforming assessment practice. Research offices are frequently asked which one to adopt.

    Framework Published Format Primary focus
    Leiden Manifesto 22 April 2015 (Nature comment) 10 principles Correct use of quantitative indicators across disciplines and settings
    DORA 2013 (San Francisco Declaration) General recommendations + signatory pledge Eliminating journal impact factor as a proxy for article or researcher quality
    CoARA 2022 (Agreement on Reforming Research Assessment) Institutional commitment agreement Sector-wide reform of hiring, promotion and funding assessment criteria

    DORA has been signed by more than 27,000 individuals and organisations, according to DORA’s own published tally as of March 2026, making it the higher-profile pledge. But when Loughborough University’s LIS-Bibliometrics committee chose a framework for its own policy in 2018, policy manager Elizabeth Gadd selected the Leiden Manifesto because it takes a “broader approach to the responsible use of all bibliometrics across a range of disciplines and settings” — not only journal-level metrics. Elsevier separately announced on 14 July 2020 that it would use the manifesto’s principles to guide its CiteScore methodology.

    In the UK, the independently commissioned Metric Tide review (2015), led by James Wilsdon for the then Higher Education Funding Council for England, reached compatible conclusions and recommended metrics support, not replace, peer review within the research administration processes underpinning the Research Excellence Framework. A research office building a REF-adjacent KPI policy should treat the two as aligned, not competing, references.

    Common questions and what comes next for research offices

    Who wrote the Leiden Manifesto for Research Metrics?

    The manifesto was written by Diana Hicks, professor of public policy at Georgia Institute of Technology, and Paul Wouters, then director of CWTS at Leiden University, together with co-authors Ludo Waltman, Sarah de Rijcke and Ismael Rafols. It was published as a comment in Nature, volume 520, on 22 April 2015.

    Does the Leiden Manifesto ban the use of bibliometrics tools?

    No. The manifesto does not prohibit bibliometrics tools such as Web of Science, Scopus or Dimensions. It requires that any output from these tools — citation counts, h-indices, journal metrics — be interpreted alongside qualitative expert review and adjusted for field-specific citation norms before it informs a decision.

    Why does the importance of bibliometrics remain contested?

    Bibliometrics matter because they scale evaluation across thousands of researchers where individual peer review is impractical. The contested part is misuse: treating a single indicator as an objective proxy for quality, rather than one input alongside portfolio review, mission fit and field context, as the manifesto’s ten principles specify.

    How often should a research office review its KPI framework under the manifesto?

    Principle 10 requires indicators to be “scrutinised regularly and updated,” but sets no fixed interval. Good institutional practice, reflected in library and research-office guidance built on the manifesto, is an annual technical review of data sources plus a full policy review on the same three-to-five-year cycle as promotion-criteria revisions.

    The Leiden Manifesto’s ten principles were written as durable evaluation ethics, not a one-time compliance exercise. As institutions layer AI-assisted analytics, altmetrics and funder-mandated open-data reporting onto existing KPI frameworks, the manifesto’s core requirement — that quantitative evaluation support, not replace, expert judgement — becomes harder to satisfy by default and more important to audit deliberately. Research offices that build the checklist above into their annual promotion-criteria review cycle, rather than treating the manifesto as background reading, are the ones actually applying it.

  • OpenAlex: The Case for Open Research Metrics

    OpenAlex is a free, CC0-licensed index of more than 319 million scholarly works, authors and institutions, built by the non-profit OurResearch to replace the discontinued Microsoft Academic Graph. For institutions weighing research-metrics platforms, its open data answers a question closed commercial indices cannot: who can audit the numbers behind an assessment decision.

    OpenAlex is a bibliographic catalogue of scientific papers, authors and institutions accessible in open-access mode, named after the Library of Alexandria. That single design choice — publishing the full dataset under a public-domain licence rather than behind a subscription wall — is what separates it structurally from Elsevier’s Scopus and Clarivate’s Web of Science, and why it has become a reference point in debates about research-assessment transparency.

    What Is OpenAlex?

    OpenAlex launched in January 2022, built by OurResearch (a US non-profit operating as Impactstory, Inc.) as a successor to the Microsoft Academic Graph, which Microsoft stopped updating on 31 December 2021. The project inherited MAG’s dataset and rebuilt it as an open, queryable graph of works, authors, institutions, funders, and topics.

    Two design decisions define the platform. First, the entire dataset is released under a Creative Commons Zero (CC0) licence, meaning any institution, developer, or researcher can download, redistribute, and build on it without permission or cost. Second, OpenAlex has formally adopted the Principles of Open Scholarly Infrastructure (POSI), a governance commitment covering sustainability, community control, and data portability.

    The scale is now substantial. OpenAlex’s own catalogue reports more than 319 million scholarly works, and its API handled roughly 115 million queries a month in 2024, according to figures cited in the platform’s Wikipedia entry. It draws source data from Crossref, ORCID, DOAJ, and Unpaywall rather than from a closed editorial pipeline.

    How Does OpenAlex Compare with Scopus and Web of Science?

    The practical difference is not just price — it is what each platform lets an institution verify. Scopus and Web of Science apply proprietary, selective journal-inclusion criteria and sell access to the resulting index. OpenAlex indexes broadly by default and publishes the inclusion logic as open code, which means an institution can inspect exactly why a work is or is not counted.

    Dimension OpenAlex Scopus (Elsevier) Web of Science (Clarivate)
    Governance Non-profit (OurResearch), POSI-aligned Commercial publisher Commercial data company
    Data licence CC0, fully open, bulk download Proprietary, licensed access only Proprietary, licensed access only
    Core journal metric No proprietary journal metric CiteScore (four-year citation average) Journal Impact Factor
    Coverage approach Broad, automated aggregation, strong Diamond OA and non-English coverage Curated, selective journal list Curated, selective journal list
    Cost to institutions Free API; optional paid support tier Subscription Subscription

    CiteScore, Scopus’s flagship journal metric, averages the citations a journal’s documents receive over a four-year window — a useful signal, but one calculated entirely inside a closed system that institutions cannot independently reproduce. OpenAlex does not publish an equivalent branded journal score; instead it exposes the underlying citation and work-level data so that any bibliometrician can calculate their own indicator and show their working.

    Coverage differences matter for equity as much as accuracy. A 2024 study cited in OpenAlex’s Wikipedia entry found the platform indexes more than 12,500 Diamond Open Access journal titles, including over 60% of Diamond OA journals absent from both Web of Science and Scopus — a direct consequence of not gating inclusion behind a commercial selection committee.

    Why Does Open Metrics Infrastructure Serve DORA’s Transparency Principle?

    The San Francisco Declaration on Research Assessment (DORA), first published in 2012, asks funders, institutions, and publishers to stop substituting journal-based proxies for direct evaluation of research and to be explicit about the criteria used in funding, hiring, and promotion decisions. That explicitness requirement is where the platform choice stops being neutral.

    A closed index can tell an institution that a number was calculated a certain way, but it cannot let that institution independently verify how, because the underlying citation graph is licensed, not published. An open metadata layer removes that opacity: the same dataset an institution cites in a tenure file or a funding report can be downloaded, re-run, and checked by anyone, including the researcher being assessed.

    Adoption evidence has followed the argument. Leiden University announced in September 2023 that it would produce an open-source edition of its CWTS Leiden Ranking using OpenAlex data from 2024 onward. Sorbonne University announced in December 2023 that it was withdrawing its Scopus subscription in favour of OpenAlex. In 2024, France’s Ministry of Higher Education and Research pledged financial support to the project, describing it as “crucial open science infrastructure,” and the Arcadia Fund awarded OurResearch a $7.5 million grant explicitly to build OpenAlex into a sustainable alternative to commercial citation indices.

    • Leiden University: open-source CWTS Leiden Ranking edition built on OpenAlex data (from 2024)
    • Sorbonne University: Scopus subscription withdrawn in favour of OpenAlex (December 2023)
    • French Ministry of Higher Education and Research: financial commitment to OpenAlex as open science infrastructure (2024)
    • Arcadia Fund: $7.5 million grant to OurResearch for OpenAlex sustainability (March 2024)

    None of this means closed indices lack value; their curated selection and mature analytics tooling still suit some high-stakes evaluations. But where the explicit requirement is transparency rather than convenience, an auditable, CC0-licensed data layer meets DORA’s stated principle more directly than a licensed black box.

    Common Questions About OpenAlex

    What is OpenAlex used for?

    Universities, funders, and publishers use OpenAlex to track publication output, measure open-access status, benchmark institutional performance, and feed alternative rankings such as the open-source CWTS Leiden Ranking. Its free API also underpins third-party dashboards, systematic-review tools, and research-information systems that need citation and affiliation data without a subscription fee.

    Is OpenAlex legit?

    Yes. OpenAlex is maintained by OurResearch, a non-profit with a multi-year record of building open scholarly infrastructure, and it has formally adopted the Principles of Open Scholarly Infrastructure (POSI). Its data and methodology are openly licensed and auditable, and the platform is already cited in peer-reviewed scientometrics research, including a 2022 arXiv paper by its founders.

    Is OpenAlex free?

    Yes. The full dataset is released under a Creative Commons Zero (CC0) public-domain licence, and the REST API can be queried without a subscription, unlike Scopus or Web of Science. A polite-pool rate limit applies to unauthenticated use, and OurResearch offers an optional paid support tier for high-volume institutional queries.

    Who owns OpenAlex?

    OpenAlex is created and maintained by OurResearch, a US-based non-profit operating as Impactstory, Inc., not by a commercial publisher. Governance sits with a mission-driven organisation rather than a shareholder-owned company — the structural distinction that underpins its CC0 licensing and its appeal to institutions pursuing publisher-independent, DORA-aligned metrics.

    What Should Institutional Leaders Do Next?

    Platform choice is now a governance decision, not just a procurement one. An institution that cites OpenAlex data in a promotion case, a funding report, or an open-access dashboard is making a transparency claim as well as a metrics claim, and that claim should be tested before it is relied upon.

    • Map which existing assessment workflows (tenure, funding reports, rankings submissions) rely on a metric an evaluator cannot independently reproduce.
    • Pilot OpenAlex alongside — not instead of — existing subscriptions, comparing coverage gaps directly against Scopus or Web of Science outputs for your own institutional corpus.
    • Document data provenance explicitly in assessment criteria, consistent with DORA’s requirement for stated, auditable methodology.
    • Track POSI-aligned infrastructure commitments (OpenAlex, CrossRef, ORCID, ROR) as the durable layer beneath any commercial tool an institution also chooses to license.

    Open, non-proprietary metadata will not replace every function a commercial index performs today. But as funders and assessment reformers keep pressing for auditable evidence over proprietary scores, institutions that already understand — and can reproduce — their own metrics will be the ones best placed to defend them.

  • What Is Bibliometrics? A Research Office Primer

    Bibliometrics is the quantitative analysis of scholarly publications and the citations between them, used to measure research output, impact and collaboration patterns. For a research office, the practical challenge is rarely gathering these numbers — library systems, funders and university dashboards supply them constantly — but recognising which of the three main types of bibliometrics a given report represents, and what it can and cannot responsibly tell you.

    In its simplest form, bibliometrics is the statistical analysis of books, articles and other publications, most often using citation counts to describe patterns in scholarly communication. That one-line definition, drawn from the OECD’s usage and echoed by university library guides, is the starting point for everything that follows.

    What is bibliometrics?

    Bibliometrics applies statistical methods to bibliographic data — publication counts, citation counts, co-authorship networks and, increasingly, download and mention data — to describe and evaluate scholarly activity. It sits alongside scientometrics, a closely related field that extends the same statistical logic to science and technology output more broadly; in practice research offices treat the two terms as near-synonyms.

    Eugene Garfield, founder of the Institute for Scientific Information and creator of the Science Citation Index in 1964, is widely credited as a founding figure of modern bibliometrics. His citation-indexing work established the infrastructure — later commercialised as Web of Science — that most present-day bibliometric reporting still depends on.

    A metrics report a research office receives is rarely a single “bibliometric score.” It is usually a blend of three distinct analytical modes, and conflating them is the single most common source of misread reports.

    What are the three types of bibliometrics?

    Library and information science distinguishes descriptive, evaluative and relational bibliometrics. Each answers a different question, and each carries a different risk of misinterpretation when applied outside its proper scope.

    Type Core question it answers Typical output Main risk if misread
    Descriptive How much has been published, by whom, where? Publication counts, output by year, discipline or department Treated as a quality signal when it only measures volume
    Evaluative How much impact or influence has that output had? Citation counts, h-index, Journal Impact Factor, Field-Weighted Citation Impact Used to rank individuals directly, ignoring field and career-stage differences
    Relational How are researchers, topics or institutions connected? Co-authorship networks, co-citation maps, research-front clustering Read as a measure of quality rather than of structure or collaboration

    Descriptive bibliometrics is the safest category for research offices to report externally, because it counts rather than judges. Evaluative bibliometrics is the category most prone to misuse — a single h-index or Journal Impact Factor figure says nothing about an individual paper’s quality. Relational bibliometrics is the least familiar to non-specialists but the most useful for identifying emerging collaboration opportunities or research strengths across a department.

    What bibliometric indicators will appear in a metrics report?

    Most institutional metrics reports combine a handful of recurring indicators. Knowing which category each one belongs to prevents a descriptive count being read as an evaluative judgement.

    • Citation count — the raw number of times a work has been cited; evaluative, but highly field- and age-dependent.
    • h-index — an author-level figure meaning a researcher has h publications each cited at least h times; evaluative, and known to disadvantage early-career researchers and those in low-citation-rate fields.
    • Journal Impact Factor (JIF) — the average citations per article in a journal over the preceding two years; a journal-level, not an article-level, indicator.
    • Field-Weighted Citation Impact (FWCI) — a normalised indicator comparing a publication’s citations against the global average for its subject, document type and publication year; a value above 1 indicates above-average performance for that field.
    • Altmetrics — non-citation signals such as policy-document mentions, news coverage, social media activity and downloads, which supplement rather than replace citation-based evaluation.

    These indicators are drawn from different underlying databases, and coverage varies. Web of Science and Scopus apply curated, subscription-based indexing; Google Scholar offers broad, free coverage with less curation; Dimensions links publications to grants and clinical trials on a freemium basis. A report’s headline number can shift depending on which source supplied it.

    How should research offices use bibliometrics responsibly?

    Bibliometrics should inform, not replace, expert judgement. Three widely referenced frameworks set out how research offices can operationalise that principle rather than treat it as an aspiration.

    The San Francisco Declaration on Research Assessment (DORA), launched in 2012, commits signatory institutions to avoid using journal-based metrics such as the Journal Impact Factor in hiring, promotion or funding decisions. Imperial College London, for example, states it has applied this commitment since becoming a DORA signatory in 2017.

    The UK’s Metric Tide review, commissioned by the then Higher Education Funding Council for England (now part of UK Research and Innovation) and published in 2015, set out five principles for responsible metrics: robustness, humility, transparency, diversity and reflexivity. Those five principles remain the reference point most UK research offices cite when drafting internal metrics policies.

    INORMS’ Research Evaluation Working Group publishes the SCOPE framework — Start, Context, Options, Probe, Evaluate — a five-step method research administrators can apply before commissioning or interpreting any metrics report, rather than defaulting to whichever indicator a database happens to surface first.

    • Start by clarifying the purpose of the evaluation before selecting any indicator.
    • Establish the context: discipline, career stage, output type and comparator group.
    • Identify the options available, including qualitative alternatives such as peer review or narrative CVs.
    • Probe the suitability and limitations of each proposed indicator.
    • Evaluate the process itself once the assessment is complete, and refine it for next time.

    Momentum toward narrative-based assessment has also grown outside the UK: the 2022 Coalition for Advancing Research Assessment (CoARA), joined by many Horizon Europe-affiliated funders and institutions, commits signatories to reduce reliance on journal-based and output-count metrics in funding and hiring decisions.

    It is worth distinguishing bibliometrics from contributor-level attribution. Bibliometrics counts citations and outputs; it does not record who did what on a given paper. CASRAI originated the CRediT contributor role taxonomy in 2014 for that separate purpose, and the standard is now stewarded by NISO as ANSI/NISO Z39.104-2022. A research office reconciling a bibliometrics report with authorship disputes should reach for CRediT roles, not citation counts.

    Common questions about bibliometrics

    What is the meaning of bibliometric?

    “Bibliometric” describes any measure derived from the statistical analysis of published scholarly output — most commonly publication counts and citation counts. The term covers the underlying data point (a bibliometric) and the wider field that studies it (bibliometrics), and applies equally to authors, journals, institutions and individual articles.

    What is an example of bibliometrics?

    The h-index is the most commonly cited example: an author with an h-index of 20 has 20 publications that have each received at least 20 citations. Other everyday examples include a journal’s Impact Factor, a department’s annual publication count, and a co-authorship map showing collaboration between institutions.

    What is bibliometrics in simple terms?

    In simple terms, bibliometrics counts and analyses publications and citations to show how much research is being produced and how much attention it receives. It turns scattered publication records into structured evidence — useful for funding reports and CVs, but never a full substitute for reading the work itself.

    Who is the father of bibliometrics?

    Eugene Garfield (1925–2017) is widely regarded as the founding figure of bibliometrics and scientometrics. As founder of the Institute for Scientific Information, he created the Science Citation Index in 1964, establishing the citation-indexing infrastructure that underpins most bibliometric analysis conducted today.

    What this means for research offices

    A metrics report that blends descriptive, evaluative and relational bibliometrics without labelling which is which will inevitably be misread by whoever receives it next — a promotion panel, a funder, or a departmental head. Labelling each figure by type, and pairing evaluative indicators with the field-normalisation context they need, is a low-cost fix most research offices can apply immediately.

    As narrative-assessment frameworks such as CoARA and DORA gain signatories, research offices should expect bibliometric reports to sit alongside, not instead of, qualitative evidence in funding and promotion decisions. Building that dual capability now — clear metrics literacy plus a credible narrative-CV process — will matter more with each assessment cycle, not less.

  • Field-Weighted Citation Impact: Where It Fails

    Field-weighted citation impact (FWCI) is a Scopus-derived metric that divides a publication’s actual citation count by the citation count expected for similar documents in the same subject field, publication year and document type — a result of 1.0 marks the global average, above 1.0 marks above-average impact. Before an institution builds review, promotion or tenure (RPT) criteria around it, the underlying normalisation assumptions need scrutiny.

    Field-weighted citation impact is defined by Elsevier as the ratio of citations actually received by an output to the citations that would be expected based on the average for the global scientific output of the same subject field, publication year and document type. It is calculated using Scopus data and surfaced through SciVal and Pure.

    What is field-weighted citation impact?

    Field-weighted citation impact is a normalised, article-level citation metric built into Elsevier’s SciVal and Scopus platforms. It expresses how a specific output, author, or institution has been cited relative to a global benchmark of comparable publications, rather than in raw citation counts that inevitably favour older papers and citation-heavy fields such as biomedicine.

    An FWCI of 1.48 means a document has been cited 48% more than expected for its field, year and type. An FWCI of 0.6 means it has been cited 40% less than expected. Because the benchmark is fixed at 1.0 by construction, roughly half of all outputs in any given field will sit below that line — a distributional fact that is frequently lost in institutional reporting.

    How is FWCI calculated?

    The field-weighted citation impact formula is simple on its face: FWCI = actual citations received ÷ expected citations for similar documents. The “expected” figure is the average citation count for all Scopus-indexed documents sharing the same Scopus subject classification (ASJC code), publication year, and document type (article, review, conference paper, and so on).

    • A microbiology article published in 2023 that has received 20 citations, against a field average of 10 for similar 2023 microbiology articles, scores an FWCI of 2.0.
    • A humanities article with 3 citations against a field average of 2 scores an FWCI of 1.5 — a superficially similar score built on a far smaller, more volatile citation base.
    • SciVal aggregates FWCI across an author’s or institution’s full output set by summing actual citations and expected citations separately, then dividing the totals — not by averaging individual FWCI scores.

    This matters: a single highly cited outlier can lift a whole portfolio’s FWCI, which is why SciVal documentation recommends reading FWCI alongside output volume and citation distribution, not as a standalone score.

    FWCI vs CiteScore and the Journal Impact Factor

    FWCI is often confused with journal-level metrics because all three numbers look similar — a decimal hovering near 1 to 10. They measure different things at different units of analysis, which is the first source of misapplication in policy documents.

    Metric Unit of analysis Field-normalised? Source and window
    Field-weighted citation impact (FWCI) Article, author, or institution Yes — field, year, document type Scopus data via SciVal; typically a rolling multi-year citation count
    CiteScore Journal No Elsevier/Scopus; launched December 2016; citations in a year to the prior 3 years of documents
    Journal Impact Factor (JIF) Journal No Clarivate Journal Citation Reports; historically a 2-year citation window

    Neither CiteScore nor the JIF adjusts for subject field, so comparing a mathematics journal’s CiteScore to an oncology journal’s compares citation cultures, not quality. FWCI’s field normalisation is what DORA-aligned reformers have asked journal metrics to do and mostly do not — which is also why FWCI is sometimes waved through review committees as the “responsible” metric without further scrutiny.

    Where FWCI breaks down: five assumptions to scrutinise

    FWCI’s field normalisation is a genuine improvement over raw citation counts and journal-level proxies, but it inherits several assumptions that DORA-aligned institutions should test before writing it into criteria.

    • Mean-based benchmarking, not percentile-based. FWCI compares an output to the field average, but citation distributions are heavily right-skewed: a small number of highly cited papers pull the mean upward, so most papers structurally score below 1.0 even when performing typically for their field. This is precisely why the Centre for Science and Technology Studies (CWTS) at Leiden University uses percentile-based indicators, such as the share of a unit’s output in the global top 10% most-cited, in its Leiden Ranking methodology rather than a mean-normalised ratio.
    • Subject classification is assigned to journals, not articles. Scopus’s ASJC subject codes are largely applied at the source-title level. An interdisciplinary article published in a broad-scope journal inherits that journal’s field classification, which can misrepresent the “expected” citation benchmark for a genuinely cross-disciplinary piece of work.
    • Small-sample volatility. For low-citation fields (much of the humanities, parts of engineering and mathematics) or for single articles, a difference of one or two citations can swing FWCI dramatically, because the expected-citation denominator is itself small. A score of 2.0 built on 20 citations is far more stable than one built on 2.
    • Self-citation is not excluded by default. Author, institutional, and journal self-citation inflate the numerator unless a self-citation exclusion is explicitly applied — a configurable option in SciVal, but one that is easy to omit when scores are pulled into a spreadsheet for a committee.
    • A single number cannot represent research quality, originality, or societal value. FWCI measures citation uptake within a fixed window; it says nothing about methodological rigour, reproducibility, data sharing, or the qualitative judgement DORA asks assessors to exercise in its place.

    Should FWCI drive review, promotion and tenure decisions?

    The San Francisco Declaration on Research Assessment (DORA), issued in 2012, recommends that institutions not use journal-based metrics as a surrogate for the quality of individual articles, individual researchers’ contributions, or as inputs to hiring, promotion, and funding decisions. FWCI’s article-level, field-normalised design addresses DORA’s specific objection to journal-level proxies such as the JIF — but it does not exempt FWCI from DORA’s broader principle that quantitative indicators should supplement, not replace, expert reading of the work itself.

    Institutions building RPT criteria around FWCI should require committees to read the underlying subject classification applied to a candidate’s outputs, check whether self-citations are excluded, and treat single-digit-citation scores as statistically unstable rather than definitive. A candidate’s FWCI trend across a full portfolio, read alongside narrative evidence of contribution, is a materially more defensible signal than a single score cited in isolation.

    As UK Research and Innovation and equivalent funders continue to align assessment frameworks with responsible-metrics principles, institutions that document how they weight FWCI against qualitative peer judgement — rather than adopting it as a pass/fail threshold — will be better positioned to defend their research administration processes to auditors, funders, and appeals panels alike.

    Frequently asked questions

    What is the average FWCI?

    The global average FWCI is always 1.0 by mathematical construction, because the benchmark for “expected citations” is itself the average of comparable outputs. A score above 1.0 indicates above-average citation performance for that field, year, and document type; a score below 1.0 indicates below-average performance.

    How do I get my field-weighted citation impact?

    FWCI is retrieved through a SciVal subscription, where institutional users can search an author, publication set, or institution and view the FWCI directly on the metrics dashboard. Some institutions also surface FWCI through Pure, which synchronises the metric from Scopus on a scheduled basis where the integration is enabled.

    What is field-weighted citation impact ranking?

    FWCI is not itself a ranking system — it is a ratio, not a percentile or league-table position. Institutions sometimes rank authors, departments, or outputs by their FWCI scores internally, but this practice inherits all the mean-based and small-sample limitations described above and should be treated cautiously.

    Is field-weighted citation impact the same as CiteScore?

    No. FWCI operates at the article, author, or institution level and is field-normalised; CiteScore is a journal-level average citation rate with no field normalisation. A journal’s CiteScore says nothing about how any single article within it actually performed relative to its field.

    FWCI remains one of the more defensible citation metrics precisely because it was built to correct the field-blindness of journal-level indicators. Its value depends entirely on institutions applying it the way its own documentation recommends: alongside output volume, subject classification checks, and self-citation controls — not as a solitary number standing in for expert judgement in a promotion file.

  • Quantitative Indicators in Research Assessment: A Hiring and Promotion Panel Guide

    Under DORA and the CoARA Agreement, quantitative indicators such as the Journal Impact Factor and h-index must never substitute for expert peer judgement in hiring and promotion decisions — they may only inform it, applied with clarity, transparency, specificity, context and fairness, alongside a broader account of a candidate’s contributions.

    Quantitative indicators in research assessment are numerical proxies for research activity — citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetrics — used, under explicit caveats, to inform rather than replace qualitative evaluation of a researcher’s work.

    Research offices translating this principle into a hiring or promotion brief face a harder question than “which metrics are banned?” Panels need operational wording for the call, the assessor briefing and the case file. This guide sets out what DORA, the CoARA Agreement and the UK’s Forum for Responsible Research Metrics concretely require, and how to turn that into panel-ready criteria.

    Contents

    What counts as a quantitative indicator in research assessment?

    A quantitative indicator is any numerical measure derived from research outputs or activity: citation counts, the h-index, the Journal Impact Factor (JIF), field-normalised citation ratios, grant income, patent counts and altmetric mentions all qualify. None was designed to certify the quality of a single article or a single person’s contribution.

    The University of York’s policy for research evaluation using quantitative data, approved by its Research Committee in November 2017, makes the distinction explicit: indicators are informative at departmental or institutional level, but “the assessment of individual research performance using solely quantitative indicators is not supported.” That collective-versus-individual distinction is the fault line every hiring and promotion policy has to draw.

    What does DORA require for hiring and promotion panels?

    The San Francisco Declaration on Research Assessment (DORA), agreed in December 2012, states a single unambiguous prohibition that panels must apply: do not use journal-based metrics, such as the Journal Impact Factor, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions. That sentence, not a general suspicion of numbers, is DORA’s operative rule for panels.

    DORA does not ban quantitative indicators outright. Its 2024 guidance document on the responsible use of quantitative indicators, produced by a DORA task force chaired by Professor Stephen Curry and published via Zenodo, sets out five principles that must govern any indicator a panel does choose to use: be clear, be transparent, be specific, be contextual, and be fair. These are DORA’s own words. Some AI-generated summaries currently paraphrase this as “the five Cs” — clarity, context, calibration, care, credit — a mnemonic that does not appear anywhere in DORA’s published guidance; panels drafting criteria should cite DORA’s actual five principles instead.

    Applied to a panel: state which indicator is being consulted and why (clear); disclose the data source and calculation method (transparent); tie the indicator to the specific claim it supports, not a general quality judgement (specific); benchmark against discipline and career stage (contextual); and check for bias against gender, geography, career breaks or non-traditional outputs (fair).

    What does the CoARA Agreement commit panels to?

    The Coalition for Advancing Research Assessment (CoARA) launched its Agreement on Reforming Research Assessment in 2022, since signed by several hundred universities, funders, national agencies and learned societies across Europe and beyond. The Agreement sets ten commitments; its core, non-negotiable commitment is to “abandon inappropriate uses in research assessment of journal- and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index.”

    Beyond that prohibition, CoARA’s commitments push panels toward qualitative peer review as the primary method, recognition of a wider range of outputs — datasets, software, protocols, policy engagement, mentoring and open-science practice — and narrative formats such as narrative CVs that let candidates describe contributions in their own words.

    The table below compares the three frameworks a UK or European research office is most likely to be asked to reconcile.

    Framework Origin and scope Core requirement for hiring/promotion Status of quantitative indicators
    DORA Global; agreed San Francisco, December 2012 Do not use JIF as a surrogate for individual quality in hiring, promotion or funding decisions Conditional use only, governed by five principles: clear, transparent, specific, contextual, fair
    CoARA Agreement Pan-European coalition; launched 2022 Core commitment to abandon inappropriate JIF/h-index use in individual assessment Indicators permitted only to support, not replace, qualitative peer review
    Forum for Responsible Research Metrics (UK) UK sector body, stemming from The Metric Tide (Wilsdon et al., HEFCE, 2015) Institutions asked to publish a responsible-metrics statement covering hiring/promotion criteria Five dimensions: robustness, humility, transparency, diversity, reflexivity

    Translating principles into concrete panel criteria

    Principles do not write themselves into a job description. A defensible panel criteria set, translating DORA, CoARA and Forum for Responsible Research Metrics guidance into working practice, includes:

    • State in the call and case-file template that the Journal Impact Factor, h-index and journal rank will not be used as proxies for individual quality (DORA’s core recommendation).
    • Offer or require a narrative CV alongside, or instead of, a conventional publication list, so data, software, mentoring and open-science contributions are visible to assessors.
    • If citation data is used at all, require field-normalised indicators rather than raw counts, and disclose the source database in the case file.
    • Credit non-publication outputs explicitly in the assessment rubric, consistent with CoARA’s broadened-recognition commitments.
    • Brief panel members on indicator limitations before each cycle, per the Forum for Responsible Research Metrics’ “humility” dimension.
    • Record, for each case, which indicators (if any) were consulted and the specific claim they supported (DORA’s “transparent” and “specific” principles).
    • Review the criteria annually, reflecting the “reflexivity” dimension shared by the Leiden Manifesto (Hicks et al., Nature, 2015) and the Forum for Responsible Research Metrics.

    A useful complementary vocabulary for the “credit non-publication outputs” step is a structured contributor-role taxonomy. CASRAI originated the CRediT contributor role taxonomy in 2014; the standard is now stewarded by NISO as ANSI/NISO Z39.104-2022. Panels reviewing narrative CVs can use CRediT’s fourteen roles to make specific, verifiable contribution claims — distinguishing data curation from formal analysis, for example — rather than relying on author order or citation counts as a proxy for who did what.

    Frequently asked questions

    What are quantitative indicators in research assessment?

    Quantitative indicators are numerical measures of research activity, including citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetric mentions. DORA’s guidance treats them as descriptive data points requiring context, not standalone quality scores, and warns against using any single indicator in isolation.

    Does DORA allow any use of quantitative indicators in hiring and promotion?

    Yes, conditionally. DORA does not ban indicators outright; it prohibits journal-based metrics like the Journal Impact Factor as a surrogate for individual quality in hiring, promotion or funding decisions. Where indicators are used, DORA’s five principles — clear, transparent, specific, contextual, fair — must govern their application.

    What does the CoARA Agreement require of hiring and promotion panels?

    CoARA’s core commitment obliges signatories to abandon inappropriate use of journal- and publication-based metrics, particularly the Journal Impact Factor and h-index, in individual assessment. Panels must prioritise qualitative peer judgement, broaden recognised output types, and adopt formats such as narrative CVs.

    What is a narrative CV, and is it required under responsible metrics guidance?

    A narrative CV lets candidates describe significant contributions — including data, software, mentoring and open-science practice — in their own words, rather than through a publication-and-citation list. DORA and CoARA both recommend narrative formats to support qualitative review, though neither makes them a formal, binding requirement.

    Implications for research offices

    UK institutions face a specific reconciliation problem: government is considering bibliometric data as an optional component of the next Research Excellence Framework exercise, REF 2029, at discipline and institutional level, even as DORA and CoARA prohibit citation-based proxies at the level of the individual hire. Policy wording needs to keep these two scales distinct — permitting aggregate bibliometric reporting upward to funders while barring the same data from an individual case file.

    The direction of travel across DORA and CoARA signatories is consistent: fewer single-number thresholds, more disclosed and contextualised indicator use, and a growing expectation that panels can explain, in writing, which evidence supported which judgement. Research offices that build this documentation habit now, rather than waiting for a funder or auditor to ask, will find each subsequent cycle easier to defend, not harder.

  • Responsible Use of Metrics: Comparing UK University DORA Guidance

    The responsible use of metrics means applying quantitative research indicators — citation counts, field-weighted citation impact, grant income — only to inform and support expert peer judgement, never to replace it, in line with the San Francisco Declaration on Research Assessment (DORA). Cambridge, Exeter, Edinburgh and UCD have each published DORA-aligned guidance for their own institutions, but a side-by-side reading shows the four documents converge on principle and diverge sharply on governance, prescriptiveness and review discipline.

    Responsible research metrics is the umbrella term for institutional policies that constrain how bibliometric and altmetric indicators may be used in hiring, promotion and funding decisions, so that no single number is treated as a proxy for research quality.

    What does “responsible use of metrics” actually require?

    DORA’s own guidance on the responsible use of quantitative indicators sets out five criteria that any institutional policy should meet: metrics use should be clear, be transparent, be specific, be contextual, and be fair, according to sfdora.org’s published guidance document. Separately, the UK’s Forum for Responsible Research Metrics — convened by Universities UK following the 2015 Metric Tide report — frames the same territory as five R’s: robustness, humility, transparency, diversity and reflexivity.

    Every institutional statement reviewed here traces back to the same three source documents: DORA (2012), the Leiden Manifesto for Research Metrics (2015), and the Metric Tide (2015, updated as Harnessing the Metrics Tide in 2022). What differs is how each university translates those shared principles into binding local policy — and that is where the real variation, and the real risk of inconsistent practice, sits.

    Where Cambridge, Exeter, Edinburgh and UCD guidance converges

    All four institutions state unambiguously that quantitative metrics must support, not supplant, qualitative expert assessment. All four are DORA signatories and all four explicitly rule out using the Journal Impact Factor as a proxy for the quality of an individual output or researcher.

    • Metrics must be applied at the correct level of granularity — never using a journal-level or institution-level number to judge an individual.
    • Comparisons between individuals must account for career stage, career breaks and part-time working.
    • Any metric used in assessment must be disclosed in advance to the people being assessed.
    • Metrics and their underlying datasets must be periodically reviewed for continued fitness of purpose.

    Edinburgh and Exeter are also both signatories to the Coalition for Advancing Research Assessment (CoARA), which each joined in 2022, committing them additionally to phasing out inappropriate use of the h-index alongside the Journal Impact Factor.

    Where the four institutions’ guidance diverges

    Beneath the shared principles, the four documents take genuinely different institutional forms — a distinction that matters more than the principles themselves for anyone trying to replicate or benchmark a policy.

    Institution Format Governance body Distinguishing feature
    University of Cambridge High-level institutional principles, devolved Individual Schools and Faculties No single university-wide rulebook; Schools write discipline-specific DORA implementation policies
    University of Exeter Nine enumerated principles (published April 2022) Responsible Metrics Champions Group, plus a DORA Champions network in Colleges and Services Explicitly built on the UCL Principles for the responsible use of bibliometrics as its starting template
    University of Edinburgh Five numbered institutional commitments (“The University will…”) Research Policy Group (2019 approval); Research Strategy Group (2025 re-approval) Only one of the four with a published review cadence — first approved April 2019, reviewed May 2025, next review Spring 2028
    University College Dublin Single institutional statement synthesising three peer frameworks Working Group on the Responsible Use of Research Metrics, reporting to the Research, Innovation and Impact Group (RIIG) Most explicit on equality, diversity and inclusion factors — names career breaks, statutory leave and part-time working directly in the policy text

    Exeter’s document is the most technically granular of the four, naming specific indicator products — Field-Weighted Citation Impact from Scopus/SciVal, Field Citation Ratio and Relative Citation Ratio from Digital Science, Category Normalised Citation Impact from Web of Science — and warning explicitly against mixing metrics from different bibliometric providers within the same assessment exercise. Edinburgh is the most procedurally binding, with a stated review cycle and a named committee for re-approval. Cambridge is the most devolved, deliberately declining to impose a single university-wide metric policy in favour of discipline-appropriate local rules. UCD is the most EDI-forward, embedding equity language directly into its core commitments rather than treating it as a supporting principle.

    What gaps remain for institutions without a dedicated policy?

    Cambridge, Exeter, Edinburgh and UCD each have a named committee, a published document and — in Edinburgh’s case — a fixed review date. Many smaller and teaching-intensive institutions have none of this. Several UK universities that rank prominently for “responsible use of metrics” searches — including library subject guides from institutions such as Derby, Plymouth and Sunderland — publish summaries of DORA’s principles rather than institutionally approved governance statements.

    That distinction is not cosmetic. A library guide can explain what responsible metrics are; it cannot bind a promotion committee the way a document approved by a Research Policy Group, a Champions Group or an RIIG can. Institutions without a dedicated policy and a named approving body carry a structural gap: staff have no enforceable assurance that a hiring panel or REF preparation exercise will actually follow the principles a library page describes. For research administrators at smaller institutions, the practical route is not to draft new principles from scratch but to adapt an existing framework — Exeter’s document explicitly credits the UCL Principles as its own starting point, and UCD’s statement was built after reviewing three existing peer institutions’ policies, showing that adaptation, not original drafting, is the established norm.

    Answer-first Q&A on research metrics

    What are the four types of metrics used in research assessment?

    Institutional guidance, including Exeter’s, groups research indicators into institutional or discipline-level indicators (rankings, field-weighted citation impact), output-level indicators (citation counts, Journal Impact Factor, altmetrics), research activity indicators (grant income, PGR numbers) and individual-focused indicators (h-index, highly-cited rankings) — each requiring different safeguards against misuse.

    What is the use of metrics in responsible research assessment?

    Metrics provide contextual, supporting evidence alongside qualitative peer review — never a standalone verdict. Under DORA and the institutional statements reviewed here, quantitative indicators may inform hiring, promotion and funding decisions only when disclosed in advance, appropriately normalised and applied at the correct level of granularity.

    What are examples of responsible research metrics?

    Commonly cited examples include field-weighted citation impact, altmetrics, grant income and postgraduate research supervision counts, used as part of a discipline-appropriate “basket of measures” rather than in isolation. Journal Impact Factor and raw h-index are explicitly excluded as individual-level proxies by every institution examined here.

    Implications for research administrators

    For research administrators, the comparison points to a practical hierarchy of maturity: a published statement with no named governance body (the entry point most smaller institutions can reach quickly); a statement with a standing committee (Exeter’s Champions Group, UCD’s Working Group reporting into RIIG); and a statement with a fixed, published review cycle (Edinburgh’s model, next due Spring 2028). Institutions preparing for REF2029 have a direct incentive to close this gap now, since metrics played a limited but real role in informing peer review for REF2021 and several universities’ Codes of Practice explicitly reserve the right to expand that role.

    The direction of travel across the sector is unambiguous: DORA and CoARA signatory numbers continue to grow, and the Forum for Responsible Research Metrics gives every institution — large or small — a ready-made template rather than a blank page. The remaining work is not persuasion but implementation: naming a governance body, setting a review date, and publishing the document where staff undergoing assessment can actually find it.

  • Forum for Responsible Research Metrics Explained

    The Forum for Responsible Research Metrics is the UK’s national coordinating body for the responsible use of research metrics. Established in 2016 following the 2015 Metric Tide review, it advises UK higher education funding bodies on metrics use in the Research Excellence Framework (REF) and promotes alignment with DORA and CoARA principles.

    The Forum for Responsible Research Metrics is an independent, sector-wide group of UK research funders, sector bodies and infrastructure experts convened to promote fair and transparent use of quantitative indicators in research assessment. It sits alongside — but is distinct from — the global DORA declaration and the European CoARA coalition, and its guidance shapes how UK institutions design metrics policy ahead of the REF.

    What is the Forum for Responsible Research Metrics?

    The Forum for Responsible Research Metrics is a UK sector body, not a regulator. It has no statutory powers and cannot compel institutions to adopt any metric or policy. Instead it functions as an advisory and convening body, bringing together funders, universities and data-infrastructure providers to agree shared principles for using metrics responsibly in research assessment.

    Its core functions, as set out at its founding, are threefold: advise the UK higher education funding bodies on metrics use in the REF; provide advocacy and leadership on responsible metrics within the UK sector; and establish links with equivalent international initiatives. The Forum is administratively supported by Universities UK (UUK), which convenes its meetings and publishes its outputs.

    When and why was the Forum established?

    The Forum’s origins trace directly to a government-commissioned review. In 2014 the then Higher Education Funding Council for England (HEFCE) — now folded into UKRI’s Research England — commissioned an independent expert group to examine the use of metrics in research evaluation, particularly within the REF. The resulting report, The Metric Tide, was published on 9 July 2015.

    The Metric Tide concluded that metrics could be a useful adjunct to peer review but warned against their use as a substitute for expert judgement, and recommended that institutions consider signing DORA or applying its principles. It also recommended that a UK-wide body be established to advise funders on metrics use in the REF, provide sector leadership, and build international links. The Forum for Responsible Research Metrics was convened in 2016 to fulfil that recommendation.

    • 2014 — HEFCE commissions the independent metrics review
    • 9 July 2015 — The Metric Tide report is published
    • 2016 — The Forum for Responsible Research Metrics is convened, supported by Universities UK
    • 2021 — Forum advice informs metrics use across the three REF2021 assessment elements
    • December 2022 — Sector commentary (Wonkhe) calls for an expanded Forum remit, including holding data providers to account

    How does the Forum relate to DORA and CoARA?

    The Forum, DORA and CoARA are three distinct bodies with overlapping but separate mandates, and conflating them is a common source of confusion for research offices drafting policy. The table below sets out how each operates and how they connect to one another.

    Body Founded Scope Core mechanism Link to the Forum
    Forum for Responsible Research Metrics 2016 UK-wide, advisory to HE funding bodies Advises funders on metrics use in the REF; monitors DORA/CoARA uptake —
    DORA (San Francisco Declaration on Research Assessment) 2013 International, institution/publisher signatory declaration Signatories pledge not to use the Journal Impact Factor as a proxy for individual quality in hiring, promotion or funding decisions The Metric Tide recommended UK institutions sign DORA; the Forum promotes and tracks its adoption
    CoARA (Coalition for Advancing Research Assessment) 2022 Pan-European coalition, member commitments Signatories commit to reforming assessment criteria and procedures over a defined implementation period Complementary European framework; several UK signatories hold both DORA and CoARA membership alongside Forum-aligned institutional policy
    REF (Research Excellence Framework) 2014 (successor to the RAE) UK-wide, all HE institutions Periodic peer-review-led assessment of research quality, impact and environment The Forum’s core client — its advice shapes how funding bodies use metrics within REF criteria

    In practice, the Forum does not ask institutions to sign anything. DORA and CoARA are commitments an institution opts into; the Forum’s guidance is advisory input into how UK funders design and apply metrics within a statutory national exercise.

    What is the Forum’s role in the REF?

    The Forum’s most concrete, recurring function is advising UK higher education funding bodies on how metrics should — and should not — be used within the REF’s three assessment elements: outputs, impact and environment. This advice fed directly into REF2021 guidance and is expected to inform preparation for the next exercise, REF 2029.

    UKRI’s Research England states that the Forum works to improve the data infrastructure underpinning metric use and the broader culture around research metrics, not just the rules for a single assessment cycle. That distinction matters for research offices: Forum guidance is a standing reference point for metrics governance, not a one-off REF submission checklist.

    • Outputs — metrics may inform but must not substitute for peer review of research quality
    • Impact — quantitative indicators supplement, rather than replace, narrative impact case studies
    • Environment — metrics contribute contextual evidence on research culture and infrastructure

    What does the Forum’s guidance mean for research offices?

    For research administrators building or reviewing a metrics policy, Forum guidance sets a de facto national baseline: metrics should be used transparently, contextually, and never as an automatic proxy for quality in hiring, promotion or funding decisions. This mirrors DORA’s core ask but is framed specifically for REF-facing institutional practice.

    Institutional research offices typically apply this in three ways: auditing existing use of journal-level and author-level metrics in internal review processes; documenting which indicators are used for which decisions and why; and aligning local policy statements with DORA and, where relevant, CoARA commitments so REF-facing metrics governance is not developed in isolation. Institutions building or auditing research assessment policy can find related structural guidance in CASRAI’s research administration resources.

    The December 2022 sector call to expand the Forum’s remit — including holding metrics data providers to account — signals that its scope is likely to widen beyond REF-facing advice toward broader accountability for the commercial infrastructure that supplies citation and impact data. Research offices should treat current Forum guidance as a floor, not a ceiling, when drafting policy ahead of REF 2029.

    Frequently asked questions

    Who chairs the UK Forum for Responsible Research Metrics?

    As of Universities UK’s most recently published listing, the Forum is chaired by Professor Max Lu, Vice-Chancellor of the University of Surrey. The Forum itself comprises representatives from UK research funders, sector bodies and data-infrastructure organisations, convened administratively by Universities UK.

    Is the Forum for Responsible Research Metrics the same as DORA?

    No. DORA is a global signatory declaration institutions and individuals opt into; the Forum is a UK sector body advising funders on metrics within the REF. The Metric Tide review recommended UK institutions sign DORA, and the Forum monitors and promotes that uptake without being DORA itself.

    What did the Metric Tide report recommend?

    Published 9 July 2015, The Metric Tide recommended that metrics support but never replace peer review, that institutions consider signing DORA, and that a national Forum be established to advise funders on responsible metrics use in the REF and build international links.

    Does the Forum’s advice apply to REF 2029?

    Yes. The Forum’s advisory role is standing, not exercise-specific, and its guidance on metrics in outputs, impact and environment assessment is expected to inform funding-body preparation for REF 2029 as it did for REF2021.

    What’s next for responsible metrics in the UK?

    With REF 2029 preparation underway and sector pressure to widen the Forum’s remit toward data-provider accountability, UK research offices should expect Forum guidance to evolve rather than stay fixed. Institutions that align internal metrics policy with Forum principles now, rather than at the point of REF submission, will face less rework as that remit expands.