CASRAI Dictionary

Tag: quantitative indicators research assessment

  • Quantitative Indicators in Research Assessment: A Hiring and Promotion Panel Guide

    Under DORA and the CoARA Agreement, quantitative indicators such as the Journal Impact Factor and h-index must never substitute for expert peer judgement in hiring and promotion decisions — they may only inform it, applied with clarity, transparency, specificity, context and fairness, alongside a broader account of a candidate’s contributions.

    Quantitative indicators in research assessment are numerical proxies for research activity — citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetrics — used, under explicit caveats, to inform rather than replace qualitative evaluation of a researcher’s work.

    Research offices translating this principle into a hiring or promotion brief face a harder question than “which metrics are banned?” Panels need operational wording for the call, the assessor briefing and the case file. This guide sets out what DORA, the CoARA Agreement and the UK’s Forum for Responsible Research Metrics concretely require, and how to turn that into panel-ready criteria.

    Contents

    What counts as a quantitative indicator in research assessment?

    A quantitative indicator is any numerical measure derived from research outputs or activity: citation counts, the h-index, the Journal Impact Factor (JIF), field-normalised citation ratios, grant income, patent counts and altmetric mentions all qualify. None was designed to certify the quality of a single article or a single person’s contribution.

    The University of York’s policy for research evaluation using quantitative data, approved by its Research Committee in November 2017, makes the distinction explicit: indicators are informative at departmental or institutional level, but “the assessment of individual research performance using solely quantitative indicators is not supported.” That collective-versus-individual distinction is the fault line every hiring and promotion policy has to draw.

    What does DORA require for hiring and promotion panels?

    The San Francisco Declaration on Research Assessment (DORA), agreed in December 2012, states a single unambiguous prohibition that panels must apply: do not use journal-based metrics, such as the Journal Impact Factor, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions. That sentence, not a general suspicion of numbers, is DORA’s operative rule for panels.

    DORA does not ban quantitative indicators outright. Its 2024 guidance document on the responsible use of quantitative indicators, produced by a DORA task force chaired by Professor Stephen Curry and published via Zenodo, sets out five principles that must govern any indicator a panel does choose to use: be clear, be transparent, be specific, be contextual, and be fair. These are DORA’s own words. Some AI-generated summaries currently paraphrase this as “the five Cs” — clarity, context, calibration, care, credit — a mnemonic that does not appear anywhere in DORA’s published guidance; panels drafting criteria should cite DORA’s actual five principles instead.

    Applied to a panel: state which indicator is being consulted and why (clear); disclose the data source and calculation method (transparent); tie the indicator to the specific claim it supports, not a general quality judgement (specific); benchmark against discipline and career stage (contextual); and check for bias against gender, geography, career breaks or non-traditional outputs (fair).

    What does the CoARA Agreement commit panels to?

    The Coalition for Advancing Research Assessment (CoARA) launched its Agreement on Reforming Research Assessment in 2022, since signed by several hundred universities, funders, national agencies and learned societies across Europe and beyond. The Agreement sets ten commitments; its core, non-negotiable commitment is to “abandon inappropriate uses in research assessment of journal- and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index.”

    Beyond that prohibition, CoARA’s commitments push panels toward qualitative peer review as the primary method, recognition of a wider range of outputs — datasets, software, protocols, policy engagement, mentoring and open-science practice — and narrative formats such as narrative CVs that let candidates describe contributions in their own words.

    The table below compares the three frameworks a UK or European research office is most likely to be asked to reconcile.

    Framework Origin and scope Core requirement for hiring/promotion Status of quantitative indicators
    DORA Global; agreed San Francisco, December 2012 Do not use JIF as a surrogate for individual quality in hiring, promotion or funding decisions Conditional use only, governed by five principles: clear, transparent, specific, contextual, fair
    CoARA Agreement Pan-European coalition; launched 2022 Core commitment to abandon inappropriate JIF/h-index use in individual assessment Indicators permitted only to support, not replace, qualitative peer review
    Forum for Responsible Research Metrics (UK) UK sector body, stemming from The Metric Tide (Wilsdon et al., HEFCE, 2015) Institutions asked to publish a responsible-metrics statement covering hiring/promotion criteria Five dimensions: robustness, humility, transparency, diversity, reflexivity

    Translating principles into concrete panel criteria

    Principles do not write themselves into a job description. A defensible panel criteria set, translating DORA, CoARA and Forum for Responsible Research Metrics guidance into working practice, includes:

    • State in the call and case-file template that the Journal Impact Factor, h-index and journal rank will not be used as proxies for individual quality (DORA’s core recommendation).
    • Offer or require a narrative CV alongside, or instead of, a conventional publication list, so data, software, mentoring and open-science contributions are visible to assessors.
    • If citation data is used at all, require field-normalised indicators rather than raw counts, and disclose the source database in the case file.
    • Credit non-publication outputs explicitly in the assessment rubric, consistent with CoARA’s broadened-recognition commitments.
    • Brief panel members on indicator limitations before each cycle, per the Forum for Responsible Research Metrics’ “humility” dimension.
    • Record, for each case, which indicators (if any) were consulted and the specific claim they supported (DORA’s “transparent” and “specific” principles).
    • Review the criteria annually, reflecting the “reflexivity” dimension shared by the Leiden Manifesto (Hicks et al., Nature, 2015) and the Forum for Responsible Research Metrics.

    A useful complementary vocabulary for the “credit non-publication outputs” step is a structured contributor-role taxonomy. CASRAI originated the CRediT contributor role taxonomy in 2014; the standard is now stewarded by NISO as ANSI/NISO Z39.104-2022. Panels reviewing narrative CVs can use CRediT’s fourteen roles to make specific, verifiable contribution claims — distinguishing data curation from formal analysis, for example — rather than relying on author order or citation counts as a proxy for who did what.

    Frequently asked questions

    What are quantitative indicators in research assessment?

    Quantitative indicators are numerical measures of research activity, including citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetric mentions. DORA’s guidance treats them as descriptive data points requiring context, not standalone quality scores, and warns against using any single indicator in isolation.

    Does DORA allow any use of quantitative indicators in hiring and promotion?

    Yes, conditionally. DORA does not ban indicators outright; it prohibits journal-based metrics like the Journal Impact Factor as a surrogate for individual quality in hiring, promotion or funding decisions. Where indicators are used, DORA’s five principles — clear, transparent, specific, contextual, fair — must govern their application.

    What does the CoARA Agreement require of hiring and promotion panels?

    CoARA’s core commitment obliges signatories to abandon inappropriate use of journal- and publication-based metrics, particularly the Journal Impact Factor and h-index, in individual assessment. Panels must prioritise qualitative peer judgement, broaden recognised output types, and adopt formats such as narrative CVs.

    What is a narrative CV, and is it required under responsible metrics guidance?

    A narrative CV lets candidates describe significant contributions — including data, software, mentoring and open-science practice — in their own words, rather than through a publication-and-citation list. DORA and CoARA both recommend narrative formats to support qualitative review, though neither makes them a formal, binding requirement.

    Implications for research offices

    UK institutions face a specific reconciliation problem: government is considering bibliometric data as an optional component of the next Research Excellence Framework exercise, REF 2029, at discipline and institutional level, even as DORA and CoARA prohibit citation-based proxies at the level of the individual hire. Policy wording needs to keep these two scales distinct — permitting aggregate bibliometric reporting upward to funders while barring the same data from an individual case file.

    The direction of travel across DORA and CoARA signatories is consistent: fewer single-number thresholds, more disclosed and contextualised indicator use, and a growing expectation that panels can explain, in writing, which evidence supported which judgement. Research offices that build this documentation habit now, rather than waiting for a funder or auditor to ask, will find each subsequent cycle easier to defend, not harder.

  • Research Assessment Reform: Why Collective Action Beats Solo Signatories

    Research assessment reform needs collective action because hiring, promotion and funding criteria are set independently by thousands of institutions — a single university dropping journal-based metrics gains nothing if every competing institution, funder and publisher still rewards them. Recent research-on-research literature frames this explicitly as a collective action problem: individual declarations such as DORA signal intent, but only coordinated, system-wide commitments — the model CoARA is built around — actually rewrite the incentives that determine careers.

    A collective action problem in research assessment is a situation where no single institution can achieve reform on its own without risking a competitive disadvantage, so change only happens when many actors move together under a shared, verifiable commitment.

    What Is the Collective Action Problem in Research Assessment Reform?

    A 2025 study in Minerva by sociologist Alexander Rushforth, “Research Assessment Reform as Collective Action Problem,” argues that research evaluation change cannot be reduced to individual institutional choice. Rushforth traces this through the Netherlands’ national “Recognition and Rewards” initiative, formally launched in 2019 to coordinate system-wide changes in assessment practice across the Dutch science system.

    The framing matters because it shifts the diagnosis. If assessment culture were simply a matter of institutional willpower, a DORA signature would be sufficient. If it is instead a coordination failure — where no actor can safely move first — then reform requires simultaneous, mutually reinforcing commitments from institutions, funders and publishers together.

    Why Doesn’t an Individual DORA Signature Change Hiring Criteria?

    The San Francisco Declaration on Research Assessment (DORA), launched in 2012, asks signatories to stop using journal-based metrics such as the Journal Impact Factor as a proxy for the quality of individual articles or researchers. Signing carries no binding enforcement mechanism, and DORA itself has long acknowledged that the harder work begins after signature — its 2019 guidance “You’ve signed DORA, now what?” explicitly frames hiring, promotion and funding criteria as the next, unfinished step.

    Two structural problems keep that step unfinished when institutions act alone:

    • First-mover risk. An institution that stops counting journal prestige in tenure review can be undercut in recruitment and rankings by peers who have not changed, because researcher CVs are still read against metric-based expectations elsewhere.
    • Interoperability failure. Where assessment criteria diverge sharply between institutions and countries, researcher mobility suffers — a candidate assessed holistically at one university may be filtered out by a metrics-based shortlist at the next.

    Neither problem is solved by any single signature. Both require peer institutions, funders and disciplinary societies to move on a broadly shared timetable.

    How Does CoARA’s Coordinated Model Differ From Individual Declarations?

    The Coalition for Advancing Research Assessment (CoARA) was formed around the Agreement on Reforming Research Assessment, which the European Commission signed and endorsed alongside DORA on 8 November 2022. Unlike a one-off declaration, CoARA requires member organisations to commit to a shared action plan with defined milestones, reported progress and working groups that develop common tools and criteria across institutions — moving assessment reform from individual pledge to managed, collective process.

    That coordination logic was reinforced on 4 December 2025, when CoARA and DORA released a joint statement on aligning their respective reform efforts rather than running parallel, uncoordinated campaigns. Science Europe’s April 2026 position statement, “Connecting Open Science and Research Assessment Reform,” makes the same point from the funder side: it treats open science and assessment reform as “mutually reinforcing and interdependent drivers of research cultures,” explicitly a multi-actor framing rather than an institution-by-institution one.

    Dimension Individual DORA signature Coordinated (CoARA-style) commitment
    Enforcement None — declaration of intent only Action plan with milestones and reporting
    Hiring/promotion criteria Left to each institution’s own timetable Shared working groups developing common criteria
    Competitive risk to first movers High — one institution changes alone Reduced — peers move on a shared timetable
    Researcher mobility Fragmented across institutions/countries Greater interoperability of criteria sought

    What Does the Dutch “Recognition and Rewards” Case Show?

    Rushforth’s analysis of Recognition and Rewards found that the initiative succeeded in uniting support from multiple influential national stakeholders — universities, funders and academic hospitals moving together — precisely because it was designed as a coordinated, system-wide commitment rather than a set of separate institutional pledges. It also documents genuine friction: critics raised concerns about the Netherlands “going it alone” internationally, illustrating that collective action problems exist at more than one level simultaneously — within a national system, and between that system and the rest of the world.

    The OECD’s April 2026 report “Reforming Research Assessment for Better Science” reaches a parallel conclusion at the international level, describing the current reform landscape as “a collective of organisations committed to reforming the assessment of research, researchers, and research organisations” — language that treats coordination, not individual compliance, as the operative unit of change.

    Frequently Asked Questions

    Does Signing DORA Actually Change University Hiring Practices?

    Not by itself. DORA’s own post-signature guidance states that hiring, promotion and funding decisions require separate, deliberate policy changes after signature. A signature is a public commitment; rewritten criteria documents, reviewed by hiring and promotion committees, are the actual evidence of change.

    What Is CoARA and How Does It Differ From DORA?

    CoARA is a coalition of research funders, institutions, and organisations built around the 2022 Agreement on Reforming Research Assessment. Unlike DORA’s single declaration, CoARA members commit to shared action plans, working groups and reported milestones — a coordination structure rather than a one-time pledge.

    Why Is Research Assessment Reform Described as a Collective Action Problem?

    Because no institution can safely change its own assessment criteria in isolation without risking a competitive disadvantage in recruitment and rankings. Research-on-research literature, including Rushforth’s 2025 Minerva study, argues reform requires simultaneous, coordinated commitments across many independent actors.

    Can One University Move Away From Metrics Without Being Disadvantaged?

    It can, but the Netherlands’ Recognition and Rewards case shows even a coordinated national effort faced criticism for “going it alone” relative to the rest of the world. A single institution acting without peer, funder and publisher alignment faces materially higher exposure to that same risk.

    What Should Institutions Actually Do Together?

    For research administration teams, the practical implication of the collective-action framing is direct: a DORA or CoARA signature belongs on a compliance checklist next to, not instead of, three coordination-dependent actions.

    1. Confirm hiring and promotion criteria documents have actually been rewritten, not merely a signature logged in a registry.
    2. Compare criteria against peer institutions in the same discipline and country to identify where first-mover risk is concentrated.
    3. Engage through CoARA working groups or equivalent sector bodies (ARMA, EARMA, INORMS) rather than drafting new criteria in isolation.

    Reform that stops at the signature stage produces a compliance artefact, not a changed incentive structure. The evidence from both the Dutch national case and the CoARA-DORA coordination model points the same way: assessment reform moves at the speed of the slowest coordinated group, not the fastest individual signatory. Institutions that treat their own criteria rewrite as contingent on parallel movement by peers, funders and publishers are following the pattern the research-on-research literature identifies as actually working — treating reform as a shared infrastructure problem, not a personal compliance decision.

  • Open Science and Research Assessment Reform: Science Europe’s 2026 Position Statement

    Science Europe’s April 2026 position statement, “Connecting Open Science and Research Assessment Reform,” argues that open science and assessment reform are not parallel projects but mutually reinforcing drivers of the same goal: research cultures that reward quality, integrity and collaboration rather than publication volume or journal prestige. Written for research funding and performing organisations, it recommends that the two movements be planned and implemented together rather than as separate policy tracks.

    Research assessment reform is the movement to change how research, researchers and research organisations are evaluated, replacing narrow reliance on citation counts and journal impact factors with broader, context-sensitive judgement of quality and contribution.

    What does Science Europe’s statement actually say?

    Science Europe published Connecting Open Science and Research Assessment Reform on 30 April 2026, authored by Bregt Saenen and James Morris and assigned DOI 10.5281/zenodo.19886162. It is the output of a three-year collaboration between Science Europe’s Working Groups on Research Culture and Open Science, grounded in a 2024 membership survey and a scoping review of the academic literature on open science and research culture.

    The statement’s central claim is definitive: aligned assessment systems and open science principles together enable research cultures that reward quality, integrity and collaboration, strengthening research and innovation systems and supporting diverse career pathways. It is addressed to research funding organisations (RFOs) and research performing organisations (RPOs), recommending strategic alignment of their open science and assessment-reform actions rather than treating them as separate work programmes.

    How are open science and assessment reform mutually reinforcing?

    Open science broadens what counts as a valuable research contribution — data sharing, open methods, software, public engagement — beyond the traditional peer-reviewed article. Reformed assessment is what makes that broadening stick: without evaluation criteria that recognise these practices, researchers have little institutional incentive to adopt them.

    Science Europe frames reformed assessment as the structural foundation that allows open science practices to flourish, moving evaluation beyond “oversimplified proxies of excellence and flawed metrics” toward a fuller account of research quality and impact. Conversely, open science supplies the evidence base — transparent methods, shared data, verifiable outputs — that makes qualitative, context-sensitive assessment credible and auditable.

    • Recognise a wider range of outputs and activities, including datasets, software, patents and public engagement, not only journal articles.
    • Combine qualitative peer judgement with quantitative indicators, calibrated to research field and career stage.
    • Incentivise openness and collaboration directly within funding and promotion criteria, rather than treating them as optional add-ons.

    What does this mean for funder policy across Europe?

    For funders, the statement is a direct call to stop running open science mandates and assessment-reform commitments as separate policy silos. Grant criteria, reporting requirements and reviewer guidance should be edited together, so that a researcher who shares data openly or publishes preprints is not simultaneously penalised by a review panel still anchored to journal impact factor.

    This mirrors a broader European policy convergence. The OECD published its own report, Reforming research assessment for better science, on 29 April 2026 — a near-simultaneous release that signals shared momentum across intergovernmental and funder-led channels toward evidence-based, less metric-dependent evaluation.

    Initiative Lead body Key milestone Primary focus
    Connecting Open Science and Research Assessment Reform Science Europe Published 30 April 2026 Aligning funder/RPO open science and assessment policy
    Agreement on Reforming Research Assessment CoARA Finalised 20 July 2022 Shared commitments across signatory institutions
    Recommendation on Open Science UNESCO Adopted 2021 Global normative framework for open science
    Declaration on Research Assessment (DORA) DORA coalition Originated 2012 Reducing reliance on journal impact factor

    How does this align with CoARA and the wider reform movement?

    Science Europe’s statement does not stand alone. The Coalition for Advancing Research Assessment (CoARA) finalised its Agreement on Reforming Research Assessment on 20 July 2022, per the European Commission, setting a shared direction for signatory research funders, performers and associations across and beyond Europe. UNESCO’s 2021 Recommendation on Open Science supplies the underlying normative framework, explicitly naming research assessment as a “dominant barrier” to open science becoming the norm.

    Read together, these three documents show a consistent policy arc: UNESCO sets the normative case for open science; CoARA operationalises assessment reform through institutional commitments; and Science Europe’s 2026 statement provides funders and research-performing organisations with a practical rationale for sequencing the two as one connected reform, not two competing compliance burdens.

    One concrete implication sits close to CASRAI’s own history. CASRAI originated the CRediT contributor role taxonomy in 2014. The standard is now stewarded by NISO as ANSI/NISO Z39.104-2022. CRediT’s structured recognition of distinct contributor roles — data curation, software, validation, formal analysis — is precisely the kind of granular, non-authorship contribution that reformed assessment frameworks are designed to reward, giving funders and institutions an existing, adoptable mechanism rather than requiring one to be built from scratch.

    Answer-first Q&A

    What is CoARA and how does it relate to Science Europe’s statement?

    The Coalition for Advancing Research Assessment (CoARA) is an international alliance of research funders, universities and associations committed to reforming how research and researchers are evaluated. Science Europe’s 2026 statement builds directly on CoARA’s Agreement on Reforming Research Assessment, adding an explicit open science alignment layer for its member funding and performing organisations.

    What does research assessment reform mean?

    Research assessment reform means changing the criteria and processes used to evaluate research, researchers and institutions — moving away from proxies like journal impact factor toward combined qualitative and quantitative judgement that accounts for diverse outputs, career stage and disciplinary context.

    How does open science relate to research assessment reform?

    Open science and research assessment reform are interdependent: open practices such as data sharing and preprints only spread if evaluation criteria reward them, while reformed assessment needs the transparency open science provides to judge non-traditional outputs credibly. Science Europe’s statement treats them as one combined reform agenda.

    Who should act on Science Europe’s position statement?

    The statement is addressed to research funding organisations and research performing organisations — the bodies that write grant criteria, promotion policies and institutional evaluation frameworks. Research administrators, publishers and developers of assessment tools are also directly affected stakeholders.

    Implications and what happens next

    For research administrators, the practical task is auditing existing grant and promotion criteria for contradictions between open science requirements and assessment practice — for example, mandating data deposit while still scoring applicants primarily on journal-tier publication counts. Science Europe’s statement gives institutions a citable rationale for resolving that contradiction in favour of alignment.

    Expect national funders and CoARA national chapters to reference the statement as they revise assessment guidance through 2026 and 2027, alongside the OECD’s parallel findings. Institutions that wait for a single mandated template will lag; those that begin mapping open science indicators onto existing assessment criteria now will be positioned to demonstrate compliance as funder audits catch up with the policy convergence already under way.

    See CASRAI’s research administration resources for related standards context, and the CRediT contributor roles for a working example of a granular recognition framework that supports reformed assessment.

  • New Expectations and Demands from Science (OECD)

    The OECD’s new working paper, “New expectations and demands from science: Rethinking research assessment frameworks,” maps the tensions, actors, and drivers reshaping how research is judged worldwide. Published as OECD Science, Technology and Industry Working Papers, No. 2026/07, it is a companion to the OECD’s flagship policy brief “Reforming research assessment for better science” — and it does a different job: rather than prescribing solutions, it diagnoses why current assessment systems are misaligned with what science is now expected to deliver.

    New expectations and demands from science is the OECD’s shorthand for a widening gap: funders, governments, and the public increasingly expect research to be open, collaborative, and socially useful, while most assessment frameworks still reward narrow, quantitative output counts. The working paper is a system-level literature review — it identifies who holds power over assessment criteria, what forces are pushing reform, and where the friction actually sits.

    What the OECD’s companion paper actually says

    The working paper is a 43-page system-level overview, not a set of new rules. OECD (2026) states that research assessment frameworks “play a central role in shaping the priorities, direction, and culture of scientific research,” but that they have grown misaligned with “evolving policy priorities, public expectations, and new demands from science.” The core diagnosis is over-reliance on narrow performance measures — publication counts, journal impact factors, and citation metrics — which generates what the paper calls “perverse incentives and undesirable behaviours.”

    Critically, the paper argues these narrow measures systematically undervalue activities that funders and the public now expect: collaboration across disciplines and borders, open science practices, societal engagement, and direct support to policymaking. The paper does not stop at critique — it distils a set of common reform principles from a comparative review of the international literature, intended to help policymakers and institutions design frameworks that better reflect these expectations.

    How it differs from the flagship OECD report

    The two documents were released as a pair but serve distinct purposes, and treating them as interchangeable misses the news value of the working paper. The flagship document — “Reforming research assessment for better science,” OECD Policy Briefs, No. 56 — is the prescriptive, policymaker-facing output: a short brief calling for balanced, cost-effective, transparent assessment approaches “supported by open data and carefully governed use of AI.”

    “New expectations and demands from science” is the underlying evidence base. It is longer, more academic in register, and organised around system mapping rather than recommendations. Where the policy brief tells institutions what to do, the working paper explains why the system is under strain and who the competing actors are — making it the more useful read for anyone designing an institutional assessment policy rather than just citing OECD guidance.

    Assessment dimension Traditional practice (narrow metrics) OECD’s identified shift
    Research outputs recognised Publications, citation counts, journal impact factor Datasets, software, policy contributions, teaching, public engagement
    Data sources Proprietary bibliometric databases Open, interoperable data infrastructure
    Collaboration Individual authorship credit only Team science and cross-sector collaboration valued explicitly
    Societal role Largely absent from formal criteria Societal and policy impact incorporated
    Use of AI in evaluation Ad hoc, ungoverned Carefully governed, transparent use

    Mapping the actors and tensions in research assessment

    The working paper’s distinguishing contribution is its system-level actor map — it names who sets, applies, and is judged by assessment criteria, and where their interests conflict. This is the part a policy brief cannot do in a few pages.

    • Funders and governments, who set the policy priorities that assessment frameworks are meant to serve but often lag them in criteria design.
    • Universities and research institutions, which apply assessment for hiring, promotion, and tenure and are often the slowest layer to change.
    • Publishers and indexing services, whose proprietary metrics (as flagged in expert commentary on the paper, including from bibliometrics researcher Ludo Waltman of Leiden University’s CWTS) still dominate despite the OECD’s call to shift toward open alternatives.
    • Individual researchers, whose career incentives are shaped by all of the above and who bear the practical cost of misalignment.
    • Research infrastructure and standards bodies, which build the open data and interoperable systems needed to support broader, fairer assessment criteria.

    This tension between what funders say they want and what institutional reward systems actually measure is not a new observation — the Coalition for Advancing Research Assessment (CoARA), building on the European Commission’s Agreement on Reforming Research Assessment finalised 20 July 2022, has been pushing the same reform agenda for four years. What the OECD paper adds is a comparative, OECD-wide synthesis rather than a Europe-centred coalition commitment, giving research administrators outside the EU a reference point that isn’t tied to CoARA membership.

    The timing also lines up with parallel European activity: Science Europe published its own position statement, “Connecting Open Science and Research Assessment Reform,” in April 2026, arguing that open science advances and assessment reform are “mutually reinforcing and inter-dependent drivers of research cultures.” Read together, the OECD working paper and the Science Europe statement show the reform agenda converging on the same point from two different institutional angles — global policy synthesis versus funder-coalition advocacy.

    Answer-first Q&A: what people are asking

    What is the OECD’s “New expectations and demands from science” paper?

    It is OECD Science, Technology and Industry Working Paper No. 2026/07, published 29 April 2026 as a companion to the OECD’s policy brief “Reforming research assessment for better science.” It provides a system-level literature review mapping the actors, tensions, and drivers behind global research assessment reform, without itself issuing binding recommendations.

    Why does this OECD paper matter for research administrators?

    It gives research administrators and institutional leaders a non-EU-specific, evidence-based reference for redesigning hiring, promotion, and funding-review criteria. Because it maps competing actor interests explicitly, it is more useful for internal policy justification than a short recommendations-only brief.

    What does the OECD say about quantitative indicators in research assessment?

    The paper identifies over-reliance on narrow quantitative indicators — publication counts, citation metrics, journal impact factors — as the central structural problem, arguing it produces perverse incentives and undervalues collaboration, openness, and societal engagement that funders now expect.

    How does this relate to the CoARA reform agreement?

    CoARA’s Agreement on Reforming Research Assessment, finalised by the European Commission on 20 July 2022, is a European funder-and-institution coalition commitment. The OECD’s 2026 paper covers similar ground but at OECD-wide scope, functioning as an evidence synthesis rather than a signatory pledge.

    Implications and what comes next

    For research administrators, the practical takeaway is not to wait for a single global standard. The OECD paper’s actor map is a useful diagnostic tool for institutions auditing their own promotion and funding-review criteria against the gap between stated priorities (openness, societal impact, collaboration) and what is actually measured (publication counts and journal placement).

    The convergence of OECD, CoARA, and Science Europe positions in 2026 suggests assessment reform is moving from advocacy toward implementation detail — governance of AI in evaluation, and the shift away from proprietary bibliometric data, are likely to be the next flashpoints. Institutions building or revising assessment frameworks, including those documenting contributor roles through standards such as CRediT, should treat this OECD synthesis as a system-level map to check institutional policy against, not a checklist to copy verbatim. For teams working through the practical mechanics of research administration and assessment criteria, CASRAI’s research administration resources track how these standards intersect with day-to-day institutional practice.

  • OECD’s Reforming Research Assessment for Better Science: A 2026 Guide for Research Offices

    The OECD’s 2026 report, “Reforming Research Assessment for Better Science,” concludes that research assessment relying on narrow publication metrics and commercial rankings distorts research culture, and it recommends that institutions cut low-value evaluation, adopt open data infrastructures, and use AI in assessment only with caution. For research offices, the report’s six policymaker recommendations translate into concrete changes to how institutional evaluation criteria, data sourcing, and staff training are run.

    Research assessment is the systematic process of monitoring, evaluating and reviewing research inputs, processes, outputs and impacts, carried out by governments, funders, universities and publishers. The OECD reforming research assessment for better science policy brief — OECD Policy Briefs No. 56, published 29 April 2026 — sets out why that process is misaligned with how science now works, and what research-performing organisations should do about it.

    What is the OECD’s 2026 report on reforming research assessment?

    “Reforming Research Assessment for Better Science” is an OECD Policy Brief (No. 56) published on 29 April 2026 that reviews why current research-assessment practices are misaligned with the evolving nature of science, and sets out six actions for policymakers and institutions. It is accompanied by a longer evidence base, OECD Science, Technology and Industry Working Paper No. 2026/7, “New Expectations and Demands from Science: Rethinking Research Assessment Frameworks,” which maps the actors, tensions and drivers behind the reform movement.

    Both documents are credited to the OECD Directorate for Science, Technology and Innovation, with Frédéric Sgard listed as the named contact. The brief carries the persistent identifier DOI 10.1787/f6202159-en; the working paper carries DOI 10.1787/0c685800-en. Neither document proposes a single replacement metric — instead, both argue for a system-level shift in how, and how often, assessment is conducted.

    Why does the OECD say metrics-based assessment needs reform?

    The OECD argues that heavy reliance on publication counts, citation rates and journal impact factors has produced perverse incentives, including a “publish or perish” culture that rewards quantity over quality. The brief cites peer-reviewed evidence — including Fanelli (2010) on publication bias and Öztürk and Taşkın (2024) on how metric-based evaluation fuels questionable publishing — to support this conclusion.

    Three specific harms are named:

    • High-risk, high-reward research is systematically undervalued because standard indicators cannot capture long-horizon payoff.
    • Transdisciplinary and societally engaged research is poorly captured by discipline-bound, publication-and-citation frameworks.
    • Assessment volume has grown faster than institutional capacity to absorb it, creating what the OECD calls research-assessment fatigue among researchers and administrators alike, a burden previously quantified in Technopolis Group’s 2015 REF Accountability Review.

    The report is equally direct about rankings. National and global university league tables, it states, “should not be used in RA” because they rely on non-transparent proprietary methods, are biased toward STEM subjects and English-language output, and — per the UN University’s Independent Expert Group 2023 Statement on Global University Rankings — can accentuate global, regional and national inequalities.

    What alternative evaluation tools and infrastructures does the OECD recommend?

    The OECD does not prescribe one alternative framework; instead, it maps nine existing international initiatives that research offices can draw on, and it names open, non-proprietary databases such as OpenAlex and Redalyc as viable substitutes for closed commercial data sources. The report’s own comparison table — reproduced and dated below — is the clearest single reference point for institutions deciding which framework to adopt or reference in policy documents.

    Initiative Year Core contribution
    DORA (San Francisco Declaration on Research Assessment) 2012 Discourages journal-based metrics as a proxy for quality; spawned the TARA practical-tools project in 2021
    Leiden Manifesto 2015 Principles and best practice for using quantitative indicators responsibly
    INORMS Research Evaluation Group 2018 SCOPE Framework for Research Evaluation and the “More than Our Rank” initiative
    FOLEC-CLACSO 2019 Regionally specific research-assessment guidelines for Latin America
    Hong Kong Principles 2019 Minimising perverse incentives; rewarding trustworthy research practice
    Science Europe Position Statement 2020 Recommendations on research assessment processes for funders
    CoARA (Coalition for Advancing Research Assessment) 2022 Agreement on Reforming Research Assessment, with global signatories
    Barcelona Declaration 2024 Advocates open research information infrastructure
    Global Research Council RRA Working Group 2024 An 11-dimension framework for responsible research assessment

    The OECD’s own recommendation is not to pick a winner among these, but to “promote sustained dialogue” between them and to have governments recognise alignment with these emerging international principles as a criterion within cyclical institutional assessment exercises.

    What should research offices do differently?

    The report’s six policymaker actions each carry a direct operational counterpart for institutional research offices, from auditing evaluation volume to renegotiating data contracts. Research administrators reading the brief should map each national-level recommendation onto an institutional equivalent:

    • Reduce assessment volume: audit which internal reviews, reports and dashboards serve a “clearly defined objective” — and retire those that do not.
    • Diversify data sources: reduce dependency on single proprietary bibliometric platforms by testing open alternatives such as OpenAlex alongside existing subscriptions.
    • Remove rankings from internal criteria: strip commercial league-table position from promotion, tenure and internal funding-allocation rubrics.
    • Govern AI use cautiously: where AI tools are piloted in peer-review triage or portfolio analysis, require transparent, explainable models and documented human oversight rather than opaque large language models.
    • Invest in staff capacity: the brief is explicit that “guidance, training and capacity building will be key” — senior administrators, librarians and peer reviewers all need structured onboarding to new evaluation frameworks, not just a policy memo.
    • Adopt proportionate methods: match the evaluation method (summative for decisions, formative for development, or a blend) to the actual purpose of each assessment exercise.

    Institutions already engaged with CASRAI’s research administration resources will recognise these as extensions of existing responsible-metrics and open-science commitments rather than a wholesale change of direction.

    Answer-first Q&A

    What is responsible research assessment?

    Responsible research assessment refers to evaluation approaches that incentivise, reflect and reward the plural characteristics of high-quality research rather than relying on narrow proxy metrics such as journal impact factor. It combines qualitative judgement with proportionate, context-appropriate quantitative indicators, following principles set out by DORA, the Leiden Manifesto and CoARA’s 2022 Agreement.

    Why does the OECD discourage the use of rankings in research assessment?

    The OECD states that national and global rankings are marketing tools built on non-transparent proprietary data and methods that are not adapted to different institutions’ profiles or purposes. Because they are biased toward STEM subjects and English-language scholarship, their use in funding or hiring decisions can exacerbate global, regional and national inequalities rather than reflect genuine research quality.

    What role should AI play in research assessment, according to the OECD?

    The OECD says AI’s role in research assessment “needs to be carefully examined” rather than adopted by default. It favours transparent, deterministic models over opaque large language models, requires ex-ante risk assessment and human oversight, and warns that AI licensing costs can quietly increase institutions’ dependency on commercial technology providers.

    How can research offices reduce the burden of research assessment?

    Research offices can reduce burden by evaluating “only what and when necessary,” in the OECD’s words — applying assessment solely where a clearly defined objective exists and a less resource-intensive process would not suffice. Matching evaluation type (summative versus formative) to actual purpose, rather than defaulting to full review, is the report’s core proportionality test.

    What happens next for research assessment reform?

    The OECD frames reform as an iterative, long-term structural transition rather than a one-off policy change, pointing to national experiments already under way as evidence. It cites Rushforth’s 2024 analysis of the Netherlands’ “Recognition and Rewards” programme and China’s institutional hybrid responses (Liang, Zhao and Li, 2024) as examples of top-down signals interacting with bottom-up institutional experimentation.

    Concrete pilots are already generating data: Luxembourg’s National Research Fund reports three years of narrative-CV use as of 2026, and UK researchers have begun assessing generative AI’s potential role ahead of the REF 2029 exercise. For research offices, the practical takeaway is that no single framework will be mandated — institutions that start testing proportionate, criteria-linked alternatives now will be better positioned as national funders and assessment bodies converge around the OECD’s six recommendations.