On November 12, 2026, computer scientist João Paulo Papa of São Paulo State University released a reproducible bibliometric audit framework on arXiv (arXiv:2611.17233v1) demonstrating how unobserved annual citation records distort longitudinal research rankings. The study proves that casually substituting missing citation observations with zeros expands indicator eligibility by up to 150 percent and dramatically reshuffles Top-50 rankings, giving research administrators and evaluation offices an empirical warning against unverified bibliometric imputation.
What happened
Bibliometricians and research evaluation offices increasingly rely on longitudinal citation trajectories—such as annual citation growth, citation acceleration, and the revival of dormant papers—to measure research velocity. However, bibliographic profiles frequently present temporal gaps where annual citation figures are missing. Standard evaluation workflows often treat these missing entries as true zero-citation years without verification, conflating unrecorded observations with zero citations.
In ‘Assessing the Reliability of Longitudinal Bibliometric Rankings under Incomplete Citation Histories’, Papa investigated this issue across 616 Google Scholar records consolidated into 595 publication units over the 2024–2025 window. Using the Google Scholar Tracker (version 1.8.5, archived under DOI 10.5281/zenodo.24170285), Papa audited the citation histories. Only 246 publications had complete coverage, while 61 had internal gaps (73 missing-year instances bounded by observed years), 38 had leading gaps, 150 had trailing gaps, and 135 lacked usable history entirely, alongside 34 uncertain publication years and 38 chronology flags.
Papa modeled three counterfactual scenarios: Scenario A accepted only strictly observed citation counts; Scenario B imputed zeros for internal gaps only; and Scenario C imputed zeros for all missing years within valid publication intervals.
Methodology and key findings
The study evaluated three indicators: Growth (citation increase), Acceleration (change in annual growth rate), and Revival (delayed recognition). Imputing zeros created an eligibility explosion: computable publications for Growth and Acceleration jumped from 196 in Scenario A (32.9% of the portfolio) to 257 in B (+31.1%) and 490 in C (82.4% of the portfolio, a 150.0% surge). Revival eligibility rose from 131 to 177 (+35.1%).
Crucially, the analysis revealed a deceptive shared-universe paradox. Among publications computable across compared scenarios, rank correlation was perfect (Spearman rho = 1.000, Kendall tau = 1.000) because existing scores remained unchanged. Yet in complete rankings, newly eligible papers caused massive displacement. For Acceleration between Scenarios A and C, the mean absolute rank displacement was 172.13 positions (median displacement 243, maximum 294) across 196 shared publications.
Top-K list membership varied sharply: while Growth maintained identical Top-10, Top-20, and Top-50 sets, Acceleration saw Top-20 Jaccard overlap fall to 0.739 and Top-50 overlap drop to 0.563, replacing 14 papers. Revival dropped to 0.667 overlap at Top-10. High score clustering (over 94% tied scores) also meant ranking cutoffs cut directly through tie groups, making lists heavily sensitive to tie-breaking policies.
Why it matters for research practice
These findings carry immediate consequences for research assessment offices and bibliometric analysts. Many institutions harvest citation data from open or semi-structured platforms where citation histories are incomplete. Evaluating these sources requires understanding how database architectures differ, as explored in CASRAI comparisons of Google Scholar vs. Scopus and Scopus vs. Web of Science vs. OpenAlex.
When tracking longitudinal citation impact, relying on high rank correlations across common papers provides false confidence. While relative order among established papers is preserved, newly admitted papers can displace over 40 percent of a Top-50 cohort.
Under responsible assessment frameworks such as DORA and CoARA, institutions must avoid treating missing data as verified zeros. Evaluators must report eligibility denominators, audit retrieval provenance, and run scenario sensitivity checks before using longitudinal metrics in hiring, promotion, or funding allocation.
What changes for researchers and evaluators
To prevent distorted rankings in research evaluation, institutions and analysts should implement several operational safeguards:
- Report eligibility denominators: Always report the proportion of publications with computable data alongside any aggregate metric, disclosing which papers were excluded due to missing history.
- Audit temporal provenance: Document whether zero values represent verified zero-citation years or unobserved intervals, and flag records with chronological inconsistencies.
- Perform scenario sensitivity checks: Test indicators across both conservative (observed-only) and permissive imputation models before publishing rank lists.
- Disclose tie-breaking conventions: Because bibliometric metrics generate heavy score clustering, committees must define and disclose tie-breaking rules before compiling shortlists.
