CASRAI Dictionary

Tag: oecd research assessment reform

  • Limitations of Bibliometrics: DORA and CoARA

    Bibliometrics — the statistical analysis of publication and citation data — cannot reliably stand in for research quality on its own: field-specific citation practices, author self-citation, and outright metric gaming all distort single-number scores such as the h-index or Journal Impact Factor. This is the documented evidentiary basis for DORA and CoARA’s push to replace single-score evaluation with qualitative, multi-indicator assessment.

    Bibliometrics is the quantitative study of academic literature — citation counts, publication volume, and derived indices — used as a proxy for scholarly influence. The proxy breaks down whenever a single number is asked to carry the full weight of a quality judgement, which is precisely what large-scale hiring, promotion, tenure, and funding panels have done for decades.

    What is bibliometrics, and why does one score fall short?

    Bibliometric indicators — citation counts, the h-index, the Journal Impact Factor (JIF), and derived composite scores — were built for large-scale, aggregate comparisons, not for judging an individual scholar’s contribution. Bergstrom, West and Wiseman’s 2008 analysis in the Journal of Neuroscience put it plainly: quantitative metrics are poor choices for assessing an individual’s research output compared with the “gold standard” of reading the work and consulting domain experts.

    A single score compresses conflicting dimensions of scholarly value — novelty, rigour, reproducibility, societal reach — into one figure. That compression, not citation data itself, is the structural weakness reform movements target.

    How does field bias distort bibliometric comparisons?

    Citation practices vary sharply by discipline, so raw citation counts cannot be compared across fields. Mathematics and the humanities publish and cite far less frequently than biomedicine, and books and conference proceedings — the dominant outputs in many humanities and computing sub-fields — are tracked inconsistently, or not at all, by Web of Science and Scopus.

    Coverage gaps compound the bias. Indexing databases differ in subject breadth, subject depth, geographic coverage, language coverage, and how far back citation histories extend, so researchers publishing outside the Anglophone, journal-dominant core of a database are systematically under-counted. Belter’s 2015 review in PMC also notes that citation-based indicators require roughly two to three years after publication before they stabilise enough to be considered reliable — a lag that penalises early-career researchers and recent work by design.

    Why does self-citation inflate bibliometric scores?

    Self-citation — an author citing their own prior work — is a normal and often legitimate part of building on a research programme. It becomes a distortion when it is used strategically to inflate an individual’s citation count or a journal’s Impact Factor beyond what independent uptake of the work would justify.

    Clarivate’s Journal Citation Reports has, in past cycles, suppressed the calculated Impact Factor of titles found to display anomalous citation behaviour, including excessive journal self-citation and coordinated “citation stacking” arrangements between journals — a documented, database-level enforcement action against exactly this failure mode. At author level, unusually concentrated self-citation rates are one of the diagnostic flags bibliometricians use when auditing whether a headline citation figure reflects genuine external uptake or engineered inflation.

    Does field-weighted citation impact solve the problem?

    Field-weighted citation impact (FWCI) is a normalised metric — used in tools such as Scopus/SciVal — that adjusts a publication’s citation count against the average for its subject field, publication year, and document type, so that a score of 1.0 represents “as expected” performance for that context. It is a genuine improvement on raw citation counts because it corrects for the field-bias problem described above.

    FWCI does not, however, correct for self-citation gaming or database coverage gaps, and it remains a single number: it shows how a paper performed against a benchmark, not whether the research was rigorous or original. Reform frameworks treat field normalisation as a refinement of bibliometrics, not a licence to keep using any single indicator as a proxy for quality.

    What evidence underlies DORA and CoARA’s reform case?

    The San Francisco Declaration on Research Assessment (DORA), launched in 2012, explicitly recommends against using the Journal Impact Factor as a surrogate measure of the quality of individual research articles, and calls on institutions to assess research on its own merits using a range of qualitative and quantitative indicators. The Coalition for Advancing Research Assessment (CoARA), formed in 2022, builds on DORA’s diagnosis: its signatories commit to basing assessment primarily on qualitative, peer-reviewed judgement, supported by responsible — not exclusive — use of quantitative indicators, and to abandoning inappropriate use of journal- and publication-based metrics such as the JIF and h-index.

    Both build directly on the failure modes above: field bias, self-citation gaming, database coverage gaps, and the two-to-three-year reliability lag are the documented evidence, not abstract principle, behind the push for reform.

    Initiative Launched Core commitment
    DORA (San Francisco Declaration on Research Assessment) 2012 Stop using the Journal Impact Factor as a proxy for individual article or researcher quality
    Leiden Manifesto 2015 (Hicks et al., Nature 520, 429–431) Ten principles for the responsible, transparent use of quantitative indicators alongside expert judgement
    CoARA (Coalition for Advancing Research Assessment) 2022 Base assessment primarily on qualitative peer review; abandon inappropriate JIF/h-index use in hiring, promotion and funding decisions

    Answer-first questions on bibliometric limitations

    What are the main limitations of bibliometrics in research assessment?

    The main limitations are field bias (citation norms differ by discipline), database coverage gaps (books, non-English and non-journal outputs are under-tracked), self-citation inflation, and a two-to-three-year lag before citation counts stabilise. Together these mean a single score cannot substitute for expert, qualitative judgement of research quality.

    Why is the h-index considered a poor measure of individual research quality?

    The h-index rewards volume and career length over insight, cannot distinguish a highly cited author from a member of a large collaborative team, and does not account for field-specific citation norms. Bergstrom, West and Wiseman (2008) concluded that reading the work and consulting experts remains the more reliable standard for individual evaluation.

    What is the difference between DORA and CoARA?

    DORA (2012) is a signable declaration focused primarily on eliminating Journal Impact Factor misuse. CoARA (2022) is a membership coalition of funders, universities and academies that goes further, committing signatories to a broader, peer-review-centred reform agenda across hiring, promotion, and institutional evaluation, with periodic reporting on progress.

    What is a self-citation rate and why does it matter?

    A self-citation rate is the proportion of an author’s or journal’s total citations that come from their own prior work rather than independent external uptake. Bibliometricians and citation-database auditors (including Clarivate’s Journal Citation Reports process) use unusually high self-citation rates as a flag for possible metric gaming rather than genuine scholarly influence.

    What should research administrators do differently?

    For research administrators and institutional leaders, the practical implication is not to discard citation data but to stop letting any single figure carry a hiring, promotion, or funding decision unsupervised. That means:

    • Pairing field-normalised indicators such as FWCI with narrative, qualitative peer assessment, as CoARA commitments require.
    • Auditing self-citation and journal self-citation patterns before citing a headline figure in a case file.
    • Recognising a fuller range of outputs — datasets, software, policy influence — rather than journal articles alone.
    • Crediting individual contributions on multi-author papers explicitly, rather than inferring credit from author position or aggregate citation share.

    On that last point, standardised contributor-role taxonomies address a related gap directly. CASRAI originated the CRediT contributor role taxonomy in 2014; the standard is now stewarded by NISO as ANSI/NISO Z39.104-2022, and it lets institutions record which named contributor performed which specific role on a paper — conceptualisation, data curation, writing — rather than relying on citation share or author-list position as a proxy for who did what.

    Where bibliometric reform goes next

    The evidentiary case against single-number bibliometric scores is now well established: field bias, database coverage gaps, self-citation gaming, and a multi-year reliability lag are documented, auditable failure modes, not theoretical objections. DORA and CoARA translate that evidence into institutional commitments, and field-normalised metrics such as FWCI narrow — without eliminating — the field-bias problem.

    The direction of travel for funders, universities and academies is toward layered assessment: responsibly used quantitative indicators, transparent contributor-role attribution, and peer judgement at the centre, rather than any one score standing alone.

  • Quantitative Indicators in Research Assessment: A Hiring and Promotion Panel Guide

    Under DORA and the CoARA Agreement, quantitative indicators such as the Journal Impact Factor and h-index must never substitute for expert peer judgement in hiring and promotion decisions — they may only inform it, applied with clarity, transparency, specificity, context and fairness, alongside a broader account of a candidate’s contributions.

    Quantitative indicators in research assessment are numerical proxies for research activity — citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetrics — used, under explicit caveats, to inform rather than replace qualitative evaluation of a researcher’s work.

    Research offices translating this principle into a hiring or promotion brief face a harder question than “which metrics are banned?” Panels need operational wording for the call, the assessor briefing and the case file. This guide sets out what DORA, the CoARA Agreement and the UK’s Forum for Responsible Research Metrics concretely require, and how to turn that into panel-ready criteria.

    Contents

    What counts as a quantitative indicator in research assessment?

    A quantitative indicator is any numerical measure derived from research outputs or activity: citation counts, the h-index, the Journal Impact Factor (JIF), field-normalised citation ratios, grant income, patent counts and altmetric mentions all qualify. None was designed to certify the quality of a single article or a single person’s contribution.

    The University of York’s policy for research evaluation using quantitative data, approved by its Research Committee in November 2017, makes the distinction explicit: indicators are informative at departmental or institutional level, but “the assessment of individual research performance using solely quantitative indicators is not supported.” That collective-versus-individual distinction is the fault line every hiring and promotion policy has to draw.

    What does DORA require for hiring and promotion panels?

    The San Francisco Declaration on Research Assessment (DORA), agreed in December 2012, states a single unambiguous prohibition that panels must apply: do not use journal-based metrics, such as the Journal Impact Factor, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions. That sentence, not a general suspicion of numbers, is DORA’s operative rule for panels.

    DORA does not ban quantitative indicators outright. Its 2024 guidance document on the responsible use of quantitative indicators, produced by a DORA task force chaired by Professor Stephen Curry and published via Zenodo, sets out five principles that must govern any indicator a panel does choose to use: be clear, be transparent, be specific, be contextual, and be fair. These are DORA’s own words. Some AI-generated summaries currently paraphrase this as “the five Cs” — clarity, context, calibration, care, credit — a mnemonic that does not appear anywhere in DORA’s published guidance; panels drafting criteria should cite DORA’s actual five principles instead.

    Applied to a panel: state which indicator is being consulted and why (clear); disclose the data source and calculation method (transparent); tie the indicator to the specific claim it supports, not a general quality judgement (specific); benchmark against discipline and career stage (contextual); and check for bias against gender, geography, career breaks or non-traditional outputs (fair).

    What does the CoARA Agreement commit panels to?

    The Coalition for Advancing Research Assessment (CoARA) launched its Agreement on Reforming Research Assessment in 2022, since signed by several hundred universities, funders, national agencies and learned societies across Europe and beyond. The Agreement sets ten commitments; its core, non-negotiable commitment is to “abandon inappropriate uses in research assessment of journal- and publication-based metrics, in particular inappropriate uses of Journal Impact Factor (JIF) and h-index.”

    Beyond that prohibition, CoARA’s commitments push panels toward qualitative peer review as the primary method, recognition of a wider range of outputs — datasets, software, protocols, policy engagement, mentoring and open-science practice — and narrative formats such as narrative CVs that let candidates describe contributions in their own words.

    The table below compares the three frameworks a UK or European research office is most likely to be asked to reconcile.

    Framework Origin and scope Core requirement for hiring/promotion Status of quantitative indicators
    DORA Global; agreed San Francisco, December 2012 Do not use JIF as a surrogate for individual quality in hiring, promotion or funding decisions Conditional use only, governed by five principles: clear, transparent, specific, contextual, fair
    CoARA Agreement Pan-European coalition; launched 2022 Core commitment to abandon inappropriate JIF/h-index use in individual assessment Indicators permitted only to support, not replace, qualitative peer review
    Forum for Responsible Research Metrics (UK) UK sector body, stemming from The Metric Tide (Wilsdon et al., HEFCE, 2015) Institutions asked to publish a responsible-metrics statement covering hiring/promotion criteria Five dimensions: robustness, humility, transparency, diversity, reflexivity

    Translating principles into concrete panel criteria

    Principles do not write themselves into a job description. A defensible panel criteria set, translating DORA, CoARA and Forum for Responsible Research Metrics guidance into working practice, includes:

    • State in the call and case-file template that the Journal Impact Factor, h-index and journal rank will not be used as proxies for individual quality (DORA’s core recommendation).
    • Offer or require a narrative CV alongside, or instead of, a conventional publication list, so data, software, mentoring and open-science contributions are visible to assessors.
    • If citation data is used at all, require field-normalised indicators rather than raw counts, and disclose the source database in the case file.
    • Credit non-publication outputs explicitly in the assessment rubric, consistent with CoARA’s broadened-recognition commitments.
    • Brief panel members on indicator limitations before each cycle, per the Forum for Responsible Research Metrics’ “humility” dimension.
    • Record, for each case, which indicators (if any) were consulted and the specific claim they supported (DORA’s “transparent” and “specific” principles).
    • Review the criteria annually, reflecting the “reflexivity” dimension shared by the Leiden Manifesto (Hicks et al., Nature, 2015) and the Forum for Responsible Research Metrics.

    A useful complementary vocabulary for the “credit non-publication outputs” step is a structured contributor-role taxonomy. CASRAI originated the CRediT contributor role taxonomy in 2014; the standard is now stewarded by NISO as ANSI/NISO Z39.104-2022. Panels reviewing narrative CVs can use CRediT’s fourteen roles to make specific, verifiable contribution claims — distinguishing data curation from formal analysis, for example — rather than relying on author order or citation counts as a proxy for who did what.

    Frequently asked questions

    What are quantitative indicators in research assessment?

    Quantitative indicators are numerical measures of research activity, including citation counts, the h-index, Journal Impact Factor, field-normalised citation ratios and altmetric mentions. DORA’s guidance treats them as descriptive data points requiring context, not standalone quality scores, and warns against using any single indicator in isolation.

    Does DORA allow any use of quantitative indicators in hiring and promotion?

    Yes, conditionally. DORA does not ban indicators outright; it prohibits journal-based metrics like the Journal Impact Factor as a surrogate for individual quality in hiring, promotion or funding decisions. Where indicators are used, DORA’s five principles — clear, transparent, specific, contextual, fair — must govern their application.

    What does the CoARA Agreement require of hiring and promotion panels?

    CoARA’s core commitment obliges signatories to abandon inappropriate use of journal- and publication-based metrics, particularly the Journal Impact Factor and h-index, in individual assessment. Panels must prioritise qualitative peer judgement, broaden recognised output types, and adopt formats such as narrative CVs.

    What is a narrative CV, and is it required under responsible metrics guidance?

    A narrative CV lets candidates describe significant contributions — including data, software, mentoring and open-science practice — in their own words, rather than through a publication-and-citation list. DORA and CoARA both recommend narrative formats to support qualitative review, though neither makes them a formal, binding requirement.

    Implications for research offices

    UK institutions face a specific reconciliation problem: government is considering bibliometric data as an optional component of the next Research Excellence Framework exercise, REF 2029, at discipline and institutional level, even as DORA and CoARA prohibit citation-based proxies at the level of the individual hire. Policy wording needs to keep these two scales distinct — permitting aggregate bibliometric reporting upward to funders while barring the same data from an individual case file.

    The direction of travel across DORA and CoARA signatories is consistent: fewer single-number thresholds, more disclosed and contextualised indicator use, and a growing expectation that panels can explain, in writing, which evidence supported which judgement. Research offices that build this documentation habit now, rather than waiting for a funder or auditor to ask, will find each subsequent cycle easier to defend, not harder.

  • Journal Impact Factor Reform After DORA, CoARA: What the Evidence Shows

    Journal impact factor reform is the shift, led by the San Francisco Declaration on Research Assessment (DORA) and the Coalition for Advancing Research Assessment (CoARA), away from journal-level metrics as proxies for individual researcher quality in hiring, promotion and tenure. Published institutional policy audits show the shift is real but partial: peer-reviewed analysis of promotion documents finds journal impact factor (JIF) language persists at a large minority of research-intensive universities even where DORA has been signed.

    DORA is a 2012 declaration, now signed by more than 25,000 individuals and organisations worldwide, that asks institutions to stop using the JIF as a substitute measure of the quality of individual research articles in funding, appointment and promotion decisions. CoARA, launched in 2022 under the Agreement on Reforming Research Assessment (ARRA), extends the same principle into a formal European coalition with published implementation commitments and a 2029 systemic-reform deadline for signatory organisations.

    What DORA and CoARA Actually Require

    DORA and CoARA are not enforcement bodies; they are voluntary commitments that institutions self-implement. Neither has an audit or sanction mechanism of its own, which is precisely why independent policy audits — not signatory counts — are the only reliable evidence of actual practice change.

    Feature DORA CoARA
    Launched 2012 (San Francisco) 2022, under the ARRA agreement
    Core mechanism Voluntary declaration; no journal-level metrics as a proxy for article or researcher quality Ten formal commitments plus a published multi-year action plan per signatory
    Geographic base Global Predominantly European, growing globally
    Compliance check None — self-reported case studies only National chapters and working groups; EC-commissioned baseline studies
    Reported uptake 25,000+ individual and organisational signatories Around 450 European higher-education institutions, roughly 13% of the sector, per the CoARA-by-numbers uptake study

    Does Signing DORA or CoARA Actually Change Hiring Practice?

    The most direct published audit remains McKiernan et al.’s 2019 eLife meta-research study, which coded the review, promotion and tenure (RPT) documents of 129 US and Canadian universities. The findings are the clearest evidence-based answer available on this question.

    • The JIF was mentioned in RPT documents at 23% of all sampled institutions, rising to 40% among research-intensive (R-type) universities.
    • Of institutions that mentioned the JIF, 87% used it in a supportive context — encouraging its use — and none explicitly prohibited it, even where the parent university had signed DORA.
    • Where JIF appeared, 63% of mentions linked it to “quality”, 40% to impact or significance, and 20% to prestige or reputation — the exact conflation DORA was written to dismantle.

    A follow-up 2024 DORA-commissioned qualitative study of US faculty hiring and tenure assessments reached a similar conclusion: departmental practice frequently lags institutional signature, because RPT criteria are set and applied at department or faculty level, not centrally by the office that signed the declaration. This decentralisation gap is the single most consistent finding across the audit literature and is the primary reason blanket “signed DORA” claims cannot be read as evidence of changed practice.

    Which Institutions Have Implemented Reform, and How

    Where reform has taken hold, it has required a specific policy rewrite, not just a signature. Three documented examples illustrate the range of implementation depth.

    • Utrecht University (Netherlands) signed DORA in 2019 and, by 2022, had formally removed the JIF from all hiring and promotion criteria university-wide as part of its Recognition and Rewards programme, replacing it with team-science and open-science indicators — reported by Nature in July 2021.
    • University of Calgary revised its GFC Academic Staff Criteria and Processes Handbook after signing DORA, explicitly incorporating DORA’s principles into the formal criteria used in tenure and promotion committees, according to the university’s own published case study.
    • CoARA national chapters, including Spain’s, have published sector-wide mapping reports tracking which member institutions have moved from commitment to documented policy change, rather than relying on signature counts alone.

    At the coalition level, the European Commission’s independently commissioned ARRA baseline study — published in mid-2026 — found that engagement is broad and growing but that implementation progress across signatory organisations is uneven and constrained by limited institutional capacity, particularly outside research-intensive universities and outside Western Europe.

    Why Reform Stalls: Barriers Identified in the Audits

    The audit literature converges on a consistent set of structural barriers, distinct from a simple lack of institutional will.

    • Decentralised authority. Central research offices sign declarations; departments and faculties write and apply RPT criteria, creating an implementation gap that persists for years.
    • No enforceable alternative metric. DORA and CoARA prescribe what institutions should stop doing more clearly than what should replace it, leaving evaluators to fall back on familiar journal-based shortcuts under time pressure.
    • Reviewer and panel habit. External referees and appointment panels — often from non-signatory institutions — continue to reference journal prestige informally, even where the host institution’s written policy is silent on the JIF.
    • Absence of external audit. Because neither DORA nor CoARA verifies signatory compliance, self-reported case studies dominate the evidence base, which is why the McKiernan-style document audit remains the field’s methodological benchmark.

    Common Questions on DORA, CoARA and Impact Factor Reform

    Has signing DORA actually changed hiring practice at universities?

    Partially. Published audits of promotion and tenure documents show the journal impact factor still appears at roughly a quarter to two-fifths of institutions, including many DORA signatories, because departmental criteria often lag the institutional-level declaration by years.

    What is the difference between DORA and CoARA?

    DORA is a global, individually signed declaration with no formal implementation mechanism, launched in 2012. CoARA is a European-centred coalition with ten binding commitments and a published multi-year action plan per signatory institution, launched in 2022.

    Do DORA-signatory universities still use the impact factor in tenure decisions?

    Some do. A 2019 eLife audit found 87% of institutions that mentioned the JIF in tenure documents used it in a supportive, encouraging context, regardless of DORA status, showing that signature alone does not remove journal metrics from evaluation practice.

    What replaces the impact factor under responsible research assessment reform?

    Reformed institutions typically adopt narrative CVs, qualitative peer review, and broadened output categories — datasets, software, mentorship, open-science contributions — rather than a single quantitative substitute metric, per CoARA’s core commitments.

    Implications for Research Administrators and Institutional Leaders

    For research administrators, the audit evidence carries a practical conclusion: a DORA or CoARA signature is a governance commitment, not a completed policy change. Verifying reform therefore requires the same document-level audit McKiernan’s team used — checking actual RPT, hiring and promotion wording for JIF language — rather than relying on signatory-list membership as a proxy for compliance.

    Institutions serious about research assessment governance should treat departmental RPT criteria, external referee guidance and appointment panel training as the three concrete levers that determine whether a declaration changes behaviour. Coalition membership sets direction; department-level document rewrites are what the evidence shows actually moves practice.

    The trajectory across both audits and coalition-level reporting points toward continued, uneven reform rather than wholesale abandonment of journal metrics. CoARA’s 2029 milestone and the growing base of institutional case studies mean the evidence base for measuring real change — as opposed to signed intent — will keep expanding over the next several years.

  • DORA Signatories: A Checklist for Institutions

    DORA signatories are institutions, funders, publishers, or individuals who have formally endorsed the San Francisco Declaration on Research Assessment — a public commitment to stop treating journal-based metrics like the Journal Impact Factor as a proxy for research quality in hiring, promotion, and funding decisions. Signing is voluntary and unaudited: it obliges a signatory to publish a statement of intent, not to hit a fixed reform deadline.

    DORA (the San Francisco Declaration on Research Assessment) is a set of research-assessment reform principles, published on 13 May 2013, that discourages substituting journal-based metrics for qualitative judgement of individual research contributions. Note: this is unrelated to the EU’s Digital Operational Resilience Act, which regulates financial-sector ICT risk and happens to share the same acronym — this article covers only the research-assessment declaration.

    What Do DORA Signatories Actually Commit To?

    A DORA signatory commits to a principle, not a procedure. The declaration originated from a December 2012 meeting of the American Society for Cell Biology and was published on 13 May 2013. Its core ask is narrow and specific: do not use the Journal Impact Factor, or any journal-level metric, “as a surrogate measure of the quality of individual research articles” in hiring, promotion, or funding decisions.

    Beyond that central pledge, signatories are asked to:

    • Articulate explicit criteria for hiring, tenure, and promotion that credit scientific content over the venue of publication
    • Consider a broad range of research outputs — including datasets, software, and preprints — not only journal articles
    • Publish a statement outlining how the organisation intends to implement DORA’s principles

    DORA’s own registry records over 27,000 individual and organisational signatories across 174 countries, according to the live signer count maintained at sfdora.org. Notable organisational signatories include the seven UK Research Councils under UKRI and, since May 2020, Springer Nature — the first major research publisher to sign.

    How Does DORA’s Commitment Differ from CoARA’s?

    DORA and the Coalition for Advancing Research Assessment (CoARA) are often mentioned together, but their signatory obligations are not equivalent. DORA asks institutions to endorse a principle and publish a statement; CoARA asks institutions to commit to a structured, time-bound implementation process. CoARA’s Agreement on Reforming Research Assessment was published in July 2022 and has attracted more than 800 signing organisations, including funders, universities, and learned societies.

    Feature DORA (San Francisco Declaration) CoARA (Coalition for Advancing Research Assessment)
    Founding document Declaration published 13 May 2013 Agreement on Reforming Research Assessment, published July 2022
    Core ask Stop using journal-level metrics as a proxy for article or researcher quality Adopt 10 core commitments covering criteria, procedures, and tools for assessment
    Reporting requirement Public statement required for new organisational signatories since November 2022; no fixed deadline Action plan required within one year of signing
    Oversight Not an accrediting body; does not audit signatories Governed by a General Assembly and Steering Board; Secretariat hosted by the European Science Foundation
    Reported signatories 27,000+ individuals and organisations in 174 countries 800+ organisations

    In practice, DORA functions as a values statement institutions can sign quickly, while CoARA functions as an implementation programme with a governance structure and a submission deadline. Many organisations — including several UK universities and Science Europe members — hold both, using DORA as the founding principle and CoARA as the operational framework.

    What Should an Institution Check Before Signing DORA?

    Because DORA is unaudited, the real work happens internally, before and after the signature. Institutional leaders should treat signing as the start of a governance exercise, not the end of one.

    • Confirm HR, promotion committees, and funding panels understand what “not using the Journal Impact Factor” means in practice for their existing criteria
    • Audit current hiring, tenure, and grant-review documentation for explicit or implicit journal-name dependence
    • Draft (or update) the public statement now required for organisational signatories, per DORA’s Engagement and Outreach Policy, approved 2024
    • Decide whether to pursue CoARA membership alongside DORA, given CoARA’s one-year action-plan deadline
    • Name an internal owner for ongoing implementation — since DORA does not audit compliance, accountability has to be self-imposed

    Institutions that skip this internal groundwork risk the outcome DORA itself has flagged publicly: a signatory whose day-to-day assessment practice has not actually changed. Reform requires deliberate revision of hiring and promotion documentation, not a signature alone.

    Common Questions About DORA Signatories

    What does DORA stand for?

    DORA stands for the San Francisco Declaration on Research Assessment, a set of research-assessment reform principles published on 13 May 2013. It is unrelated to the EU’s Digital Operational Resilience Act, which regulates financial-sector ICT risk and shares the same acronym. Readers researching signatory obligations should confirm they mean the research-assessment declaration.

    What is the difference between a DORA signatory and a CoARA signatory?

    A DORA signatory commits to principles against using journal-level metrics in hiring, promotion, and funding, with no mandatory reporting deadline. A CoARA signatory commits to 10 explicit reform commitments and must submit a public action plan within one year of signing — making CoARA the more prescriptive route.

    Does DORA apply to UK research institutions?

    Yes. UKRI and each of its seven constituent Research Councils are DORA signatories in their own right, and Science Europe reported that 60% of its member organisations had signed DORA by March 2023. DORA carries no jurisdictional restriction — any UK university, funder, or publisher can sign voluntarily.

    What happens after an institution signs DORA?

    Signing is only the start: organisational signatories must publish a statement describing how they will implement DORA’s principles, under DORA’s Engagement and Outreach Policy (approved 2024). DORA does not audit signatories or revoke signatory status, so ongoing implementation depends entirely on internal institutional governance.

    What This Means for Institutional Strategy

    The practical distinction for institutional leaders is one of pace versus prescription. DORA delivers a values commitment that can be adopted in weeks and signals good faith to researchers and funders. CoARA delivers a structured reform pathway with a governance body, working groups, and a one-year deliverable, better suited to institutions ready to formalise assessment reform as a programme rather than a statement.

    Research administrators — through bodies such as ARMA, EARMA, and INORMS — increasingly treat the two as complementary rather than competing: DORA as the founding principle cited in policy documents, CoARA as the operational mechanism for tracking and reporting progress. Institutions weighing either commitment should map both sets of obligations against existing hiring, promotion, and funding criteria before signing either declaration, so that the public statement reflects a change already under way rather than a promise made in advance of it.

  • Open Science and Research Assessment Reform: Science Europe’s 2026 Position Statement

    Science Europe’s April 2026 position statement, “Connecting Open Science and Research Assessment Reform,” argues that open science and assessment reform are not parallel projects but mutually reinforcing drivers of the same goal: research cultures that reward quality, integrity and collaboration rather than publication volume or journal prestige. Written for research funding and performing organisations, it recommends that the two movements be planned and implemented together rather than as separate policy tracks.

    Research assessment reform is the movement to change how research, researchers and research organisations are evaluated, replacing narrow reliance on citation counts and journal impact factors with broader, context-sensitive judgement of quality and contribution.

    What does Science Europe’s statement actually say?

    Science Europe published Connecting Open Science and Research Assessment Reform on 30 April 2026, authored by Bregt Saenen and James Morris and assigned DOI 10.5281/zenodo.19886162. It is the output of a three-year collaboration between Science Europe’s Working Groups on Research Culture and Open Science, grounded in a 2024 membership survey and a scoping review of the academic literature on open science and research culture.

    The statement’s central claim is definitive: aligned assessment systems and open science principles together enable research cultures that reward quality, integrity and collaboration, strengthening research and innovation systems and supporting diverse career pathways. It is addressed to research funding organisations (RFOs) and research performing organisations (RPOs), recommending strategic alignment of their open science and assessment-reform actions rather than treating them as separate work programmes.

    How are open science and assessment reform mutually reinforcing?

    Open science broadens what counts as a valuable research contribution — data sharing, open methods, software, public engagement — beyond the traditional peer-reviewed article. Reformed assessment is what makes that broadening stick: without evaluation criteria that recognise these practices, researchers have little institutional incentive to adopt them.

    Science Europe frames reformed assessment as the structural foundation that allows open science practices to flourish, moving evaluation beyond “oversimplified proxies of excellence and flawed metrics” toward a fuller account of research quality and impact. Conversely, open science supplies the evidence base — transparent methods, shared data, verifiable outputs — that makes qualitative, context-sensitive assessment credible and auditable.

    • Recognise a wider range of outputs and activities, including datasets, software, patents and public engagement, not only journal articles.
    • Combine qualitative peer judgement with quantitative indicators, calibrated to research field and career stage.
    • Incentivise openness and collaboration directly within funding and promotion criteria, rather than treating them as optional add-ons.

    What does this mean for funder policy across Europe?

    For funders, the statement is a direct call to stop running open science mandates and assessment-reform commitments as separate policy silos. Grant criteria, reporting requirements and reviewer guidance should be edited together, so that a researcher who shares data openly or publishes preprints is not simultaneously penalised by a review panel still anchored to journal impact factor.

    This mirrors a broader European policy convergence. The OECD published its own report, Reforming research assessment for better science, on 29 April 2026 — a near-simultaneous release that signals shared momentum across intergovernmental and funder-led channels toward evidence-based, less metric-dependent evaluation.

    Initiative Lead body Key milestone Primary focus
    Connecting Open Science and Research Assessment Reform Science Europe Published 30 April 2026 Aligning funder/RPO open science and assessment policy
    Agreement on Reforming Research Assessment CoARA Finalised 20 July 2022 Shared commitments across signatory institutions
    Recommendation on Open Science UNESCO Adopted 2021 Global normative framework for open science
    Declaration on Research Assessment (DORA) DORA coalition Originated 2012 Reducing reliance on journal impact factor

    How does this align with CoARA and the wider reform movement?

    Science Europe’s statement does not stand alone. The Coalition for Advancing Research Assessment (CoARA) finalised its Agreement on Reforming Research Assessment on 20 July 2022, per the European Commission, setting a shared direction for signatory research funders, performers and associations across and beyond Europe. UNESCO’s 2021 Recommendation on Open Science supplies the underlying normative framework, explicitly naming research assessment as a “dominant barrier” to open science becoming the norm.

    Read together, these three documents show a consistent policy arc: UNESCO sets the normative case for open science; CoARA operationalises assessment reform through institutional commitments; and Science Europe’s 2026 statement provides funders and research-performing organisations with a practical rationale for sequencing the two as one connected reform, not two competing compliance burdens.

    One concrete implication sits close to CASRAI’s own history. CASRAI originated the CRediT contributor role taxonomy in 2014. The standard is now stewarded by NISO as ANSI/NISO Z39.104-2022. CRediT’s structured recognition of distinct contributor roles — data curation, software, validation, formal analysis — is precisely the kind of granular, non-authorship contribution that reformed assessment frameworks are designed to reward, giving funders and institutions an existing, adoptable mechanism rather than requiring one to be built from scratch.

    Answer-first Q&A

    What is CoARA and how does it relate to Science Europe’s statement?

    The Coalition for Advancing Research Assessment (CoARA) is an international alliance of research funders, universities and associations committed to reforming how research and researchers are evaluated. Science Europe’s 2026 statement builds directly on CoARA’s Agreement on Reforming Research Assessment, adding an explicit open science alignment layer for its member funding and performing organisations.

    What does research assessment reform mean?

    Research assessment reform means changing the criteria and processes used to evaluate research, researchers and institutions — moving away from proxies like journal impact factor toward combined qualitative and quantitative judgement that accounts for diverse outputs, career stage and disciplinary context.

    How does open science relate to research assessment reform?

    Open science and research assessment reform are interdependent: open practices such as data sharing and preprints only spread if evaluation criteria reward them, while reformed assessment needs the transparency open science provides to judge non-traditional outputs credibly. Science Europe’s statement treats them as one combined reform agenda.

    Who should act on Science Europe’s position statement?

    The statement is addressed to research funding organisations and research performing organisations — the bodies that write grant criteria, promotion policies and institutional evaluation frameworks. Research administrators, publishers and developers of assessment tools are also directly affected stakeholders.

    Implications and what happens next

    For research administrators, the practical task is auditing existing grant and promotion criteria for contradictions between open science requirements and assessment practice — for example, mandating data deposit while still scoring applicants primarily on journal-tier publication counts. Science Europe’s statement gives institutions a citable rationale for resolving that contradiction in favour of alignment.

    Expect national funders and CoARA national chapters to reference the statement as they revise assessment guidance through 2026 and 2027, alongside the OECD’s parallel findings. Institutions that wait for a single mandated template will lag; those that begin mapping open science indicators onto existing assessment criteria now will be positioned to demonstrate compliance as funder audits catch up with the policy convergence already under way.

    See CASRAI’s research administration resources for related standards context, and the CRediT contributor roles for a working example of a granular recognition framework that supports reformed assessment.

  • New Expectations and Demands from Science (OECD)

    The OECD’s new working paper, “New expectations and demands from science: Rethinking research assessment frameworks,” maps the tensions, actors, and drivers reshaping how research is judged worldwide. Published as OECD Science, Technology and Industry Working Papers, No. 2026/07, it is a companion to the OECD’s flagship policy brief “Reforming research assessment for better science” — and it does a different job: rather than prescribing solutions, it diagnoses why current assessment systems are misaligned with what science is now expected to deliver.

    New expectations and demands from science is the OECD’s shorthand for a widening gap: funders, governments, and the public increasingly expect research to be open, collaborative, and socially useful, while most assessment frameworks still reward narrow, quantitative output counts. The working paper is a system-level literature review — it identifies who holds power over assessment criteria, what forces are pushing reform, and where the friction actually sits.

    What the OECD’s companion paper actually says

    The working paper is a 43-page system-level overview, not a set of new rules. OECD (2026) states that research assessment frameworks “play a central role in shaping the priorities, direction, and culture of scientific research,” but that they have grown misaligned with “evolving policy priorities, public expectations, and new demands from science.” The core diagnosis is over-reliance on narrow performance measures — publication counts, journal impact factors, and citation metrics — which generates what the paper calls “perverse incentives and undesirable behaviours.”

    Critically, the paper argues these narrow measures systematically undervalue activities that funders and the public now expect: collaboration across disciplines and borders, open science practices, societal engagement, and direct support to policymaking. The paper does not stop at critique — it distils a set of common reform principles from a comparative review of the international literature, intended to help policymakers and institutions design frameworks that better reflect these expectations.

    How it differs from the flagship OECD report

    The two documents were released as a pair but serve distinct purposes, and treating them as interchangeable misses the news value of the working paper. The flagship document — “Reforming research assessment for better science,” OECD Policy Briefs, No. 56 — is the prescriptive, policymaker-facing output: a short brief calling for balanced, cost-effective, transparent assessment approaches “supported by open data and carefully governed use of AI.”

    “New expectations and demands from science” is the underlying evidence base. It is longer, more academic in register, and organised around system mapping rather than recommendations. Where the policy brief tells institutions what to do, the working paper explains why the system is under strain and who the competing actors are — making it the more useful read for anyone designing an institutional assessment policy rather than just citing OECD guidance.

    Assessment dimension Traditional practice (narrow metrics) OECD’s identified shift
    Research outputs recognised Publications, citation counts, journal impact factor Datasets, software, policy contributions, teaching, public engagement
    Data sources Proprietary bibliometric databases Open, interoperable data infrastructure
    Collaboration Individual authorship credit only Team science and cross-sector collaboration valued explicitly
    Societal role Largely absent from formal criteria Societal and policy impact incorporated
    Use of AI in evaluation Ad hoc, ungoverned Carefully governed, transparent use

    Mapping the actors and tensions in research assessment

    The working paper’s distinguishing contribution is its system-level actor map — it names who sets, applies, and is judged by assessment criteria, and where their interests conflict. This is the part a policy brief cannot do in a few pages.

    • Funders and governments, who set the policy priorities that assessment frameworks are meant to serve but often lag them in criteria design.
    • Universities and research institutions, which apply assessment for hiring, promotion, and tenure and are often the slowest layer to change.
    • Publishers and indexing services, whose proprietary metrics (as flagged in expert commentary on the paper, including from bibliometrics researcher Ludo Waltman of Leiden University’s CWTS) still dominate despite the OECD’s call to shift toward open alternatives.
    • Individual researchers, whose career incentives are shaped by all of the above and who bear the practical cost of misalignment.
    • Research infrastructure and standards bodies, which build the open data and interoperable systems needed to support broader, fairer assessment criteria.

    This tension between what funders say they want and what institutional reward systems actually measure is not a new observation — the Coalition for Advancing Research Assessment (CoARA), building on the European Commission’s Agreement on Reforming Research Assessment finalised 20 July 2022, has been pushing the same reform agenda for four years. What the OECD paper adds is a comparative, OECD-wide synthesis rather than a Europe-centred coalition commitment, giving research administrators outside the EU a reference point that isn’t tied to CoARA membership.

    The timing also lines up with parallel European activity: Science Europe published its own position statement, “Connecting Open Science and Research Assessment Reform,” in April 2026, arguing that open science advances and assessment reform are “mutually reinforcing and inter-dependent drivers of research cultures.” Read together, the OECD working paper and the Science Europe statement show the reform agenda converging on the same point from two different institutional angles — global policy synthesis versus funder-coalition advocacy.

    Answer-first Q&A: what people are asking

    What is the OECD’s “New expectations and demands from science” paper?

    It is OECD Science, Technology and Industry Working Paper No. 2026/07, published 29 April 2026 as a companion to the OECD’s policy brief “Reforming research assessment for better science.” It provides a system-level literature review mapping the actors, tensions, and drivers behind global research assessment reform, without itself issuing binding recommendations.

    Why does this OECD paper matter for research administrators?

    It gives research administrators and institutional leaders a non-EU-specific, evidence-based reference for redesigning hiring, promotion, and funding-review criteria. Because it maps competing actor interests explicitly, it is more useful for internal policy justification than a short recommendations-only brief.

    What does the OECD say about quantitative indicators in research assessment?

    The paper identifies over-reliance on narrow quantitative indicators — publication counts, citation metrics, journal impact factors — as the central structural problem, arguing it produces perverse incentives and undervalues collaboration, openness, and societal engagement that funders now expect.

    How does this relate to the CoARA reform agreement?

    CoARA’s Agreement on Reforming Research Assessment, finalised by the European Commission on 20 July 2022, is a European funder-and-institution coalition commitment. The OECD’s 2026 paper covers similar ground but at OECD-wide scope, functioning as an evidence synthesis rather than a signatory pledge.

    Implications and what comes next

    For research administrators, the practical takeaway is not to wait for a single global standard. The OECD paper’s actor map is a useful diagnostic tool for institutions auditing their own promotion and funding-review criteria against the gap between stated priorities (openness, societal impact, collaboration) and what is actually measured (publication counts and journal placement).

    The convergence of OECD, CoARA, and Science Europe positions in 2026 suggests assessment reform is moving from advocacy toward implementation detail — governance of AI in evaluation, and the shift away from proprietary bibliometric data, are likely to be the next flashpoints. Institutions building or revising assessment frameworks, including those documenting contributor roles through standards such as CRediT, should treat this OECD synthesis as a system-level map to check institutional policy against, not a checklist to copy verbatim. For teams working through the practical mechanics of research administration and assessment criteria, CASRAI’s research administration resources track how these standards intersect with day-to-day institutional practice.