A new content analysis published in Scientometrics in 2026 puts a number on something many observers of research assessment reform have suspected anecdotally: institutions signing on to the Coalition for Advancing Research Assessment (CoARA) talk about open science far more than they operationalize it. The study, "Advancing open science through research assessment reform? Content analysis of CoARA action plans" by Alexander Rushforth, Nino Gogadze, Tea Skhirtladze, and Janne Pölönen (Scientometrics, vol. 131, pp. 3897–3918), is a targeted linguistic audit of CoARA action plans — not a broad audit of every CoARA commitment, and not a claim about the full signatory base, which now runs into the hundreds. It is narrower and more useful than that: a systematic count of how open-science terminology actually appears across a defined, citable sample of action plan documents.
What the study actually did
The authors drew on 248 publicly available action plans submitted by CoARA signatories across 35 countries and 19 trans-national organizations. Universities dominate the sample (66%, N=164), followed by research centres and funders (10.5% each, N=26), academies and learned societies (6.5%, N=16), research assessment agencies (3.6%, N=9), and other non-profits (2.8%, N=7). Because CoARA membership still skews heavily European, the dataset mostly reflects European institutional contexts — a limitation the study itself flags, and one worth carrying over into how the findings get cited elsewhere.
Rather than reading each plan qualitatively line by line, the researchers coded mentions of open-science terms into three tiers: values terms (aspirational language — openness, transparency, equity), umbrella terms (the labels institutions use for the concept itself, e.g. "open science" vs. "open research"), and operational terms (concrete practices — things like data deposit, preregistration, or specific infrastructure). That three-tier coding scheme is the paper’s methodological contribution, and it’s what lets the results move beyond a simple word count into a diagnosis of what kind of language dominates.
The headline finding: an implementation gap, quantified
Two results stand out. First, on terminology alone: "Open Science" is by far the dominant umbrella term, appearing in 167 of the 248 plans (68%), compared to "Open Research" (20%) and "Open Scholarship" (just 2%) — a striking degree of convergence on a single label, even across 35 countries and multiple institution types.
Second, and more consequential for anyone assessing whether CoARA commitments are translating into practice: the study finds a skewed distribution across the three term tiers. Most individual open-science terms appear in fewer than 20% of the documents, and among the terms that do appear most often, values terms substantially outnumber operational terms. In plainer language, institutions are considerably more likely to declare that they value openness, transparency, or equity than to specify what they will actually do — which repository, which preregistration requirement, which concrete metric they will retire or adopt. The authors read this as evidence that CoARA action plans, at least as of this sample, emphasize aspirational principle over operational commitment.
That finding sits alongside — and gives empirical texture to — a broader pattern already visible in reporting on CoARA’s own ten commitments: a coalition can grow its signatory base quickly while implementation quality and pace vary widely by institution. CASRAI has covered that variation from the funding side (see the MA3 Challenge coverage of the move from declaration to implementation) and the national-chapter side (Italy, the UK, Ireland). This paper is the first systematic, document-level content analysis CASRAI has flagged that puts an actual percentage on the values-versus-operational gap, rather than relying on case-by-case institutional reporting.
Why the framing matters
It’s worth being precise about scope, because this exact study is easy to overstate. It is not an audit of the roughly 800+ organizations that have now signed the CoARA Agreement — it covers the 248 action plans that were publicly available for coding at the time of the study, a meaningfully smaller and earlier-adopting subset. It also isn’t a judgment on whether any individual institution is acting in good faith; a plan heavy on values language in year one or two of a multi-year commitment isn’t necessarily a plan that will stay that way. What the paper does establish, with a transparent and replicable method, is a baseline: as of this sample, the language institutions use to describe their reform commitments leans aspirational, and concrete operational detail is comparatively rare.
That distinction matters for how the finding should be used. It’s a legitimate data point for research offices benchmarking their own action plan against peers, for CoARA’s own working groups tracking maturation over time, and for anyone building on CASRAI’s existing DORA vs. CoARA comparison or the broader publishing and research-assessment cluster. It is not evidence that CoARA reform is failing, and coverage that flattens "values terms outnumber operational terms in a 248-document content analysis" into "CoARA isn’t working" would be overreading a careful, narrowly scoped study.
What to watch next
The natural follow-up — which the authors’ framing invites even if this particular paper doesn’t attempt it — is a second wave: does the values/operational ratio shift as more signatories move from first-generation to revised action plans, and does it differ systematically by institution type (funders vs. universities vs. academies) or by national chapter? CASRAI’s national-chapter coverage suggests real variation in how CoARA commitments cash out locally; a longitudinal version of this content-analysis method would be one of the more rigorous ways to test whether that variation is narrowing or widening as the coalition matures past its first few years.
