Introduction
For decades, academic hiring, promotion, and funding decisions have heavily relied on simplistic, journal-level quantitative metrics, primarily the Journal Impact Factor (JIF) and h-index. This over-reliance has created perverse incentives, encouraging quantity over quality, scientific conformity over breakthrough risk-taking, and hyper-competitiveness that devalues collaborative and open science.
The San Francisco Declaration on Research Assessment (DORA)
Drafted in 2012, the San Francisco Declaration on Research Assessment (DORA) was a milestone in challenging this metric-centric evaluation culture. Its primary recommendation is simple yet revolutionary: do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles to assess an individual scientist’s contributions.
The Coalition for Advancing Research Assessment (CoARA)
Building on DORA, the Coalition for Advancing Research Assessment (CoARA), launched in 2022, represents a systematic global coalition to reform evaluation systems. CoARA establishes a shared direction based on ten commitments, emphasizing qualitative judgment with peer-review at its core, supported by the responsible use of quantitative indicators, and respecting the diversity of research outputs.
Practical Institutional Pathways for Reform
Transitioning to responsible evaluation requires concrete policy changes within universities. This includes: 1. Adopting Narrative CV formats (like the Resume for Researchers) where academics describe their contributions contextually. 2. Training review panels on the limitations of bibliometrics. 3. Rewarding open science, data curation, public outreach, and teaching contributions alongside publications.
Key Comparison Matrix
| Evaluation Model | Key Characteristics | Main Advantages | Hurdles to Adoption |
|---|---|---|---|
| Metric-Centric | Heavy reliance on JIF, h-index, and citation counts. | Quick, low administrative overhead, seemingly objective. | Encourages citation gaming, devalues non-article outputs. |
| Responsible/Holistic | Peer review, qualitative CVs, and targeted responsible metrics. | Recognizes diverse contributions, rewards open science. | Higher panel review time, requires cultural shift. |
Actionable Checklist for Responsible Assessment
- Incorporate DORA principles explicitly into university promotion and tenure guidelines.
- Introduce narrative-style CV templates for internal institutional grant proposals.
- Explicitly prohibit the mention of Journal Impact Factors in job descriptions and promotion files.
- Provide bibliometric education sessions for hiring and promotion committees.
- Define metrics policies that reward software, dataset publications, and mentoring activities.
Leave a Reply