On November 12, 2026, researchers from the University of Cambridge, the Flatiron Institute, and Universitat Autònoma de Barcelona released an arXiv preprint auditing AI peer reviewers against 357 objective errors injected into real scientific manuscripts. The benchmark reveals that automated reviewer performance varies by up to 50 percentage points based on model selection and prompting, providing journal editors and research integrity teams with empirical criteria before deploying automated screening tools.
What happened
In their preprint titled “Auditing AI peer reviewers: a benchmark on real scientific papers” (arXiv:2611.17052v1), authors Íñigo Zubeldia, Boris Bolliet, Francisco Villaescusa-Navarro, Pablo Villanueva-Domingo, and Miles Cranmer established a standardized evaluation framework to quantify how effectively large language models detect substantive defects in academic manuscripts. Rather than relying on subjective reviewer satisfaction or alignment with human peer reviews, the team introduced a perturbation-based protocol that inserts objective, localisable flaws into real research papers.
The evaluation corpus comprised 30 recent arXiv preprints published after the knowledge cut-off dates of all tested models: 15 manuscripts in astrophysics from astro-ph (median 13 pages) and 15 in machine learning from cs.LG (median 9 pages). Into these papers, the researchers injected 357 concrete errors across four distinct categories: general logical and cross-section inconsistencies, mathematical derivations, numerical calculations, and bibliographic citations. Each injected error was calibrated across three severity levels: subtle (S1), clear (S2), and blatant (S3). To keep per-copy defect density representative of normal scientific publishing, the 12 error combinations per paper were distributed across three separate copies alongside an untouched clean manuscript used to measure false-positive rates.
Methodology and key findings
The authors evaluated 12 reviewer systems spanning three architectural paradigms: single-LLM zero-shot reviewers (Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro Preview), an agentic coding environment (Claude Code running with bash, Python, and computer-algebra tooling), three public external baselines (the Stanford Agentic Reviewer, OpenReviewer-8B, and PaperAudit-Bench Deep), and Skepthical, a dedicated multi-agent audit framework designed by the authors that combines cross-family models with automated reference retrieval, SymPy symbolic math verification, and Python numerical execution.
The benchmark produced several critical findings for academic publishing and editorial administration:
- Model and prompt choice dominate baseline performance: When presented with an identical one-sentence prompt, frontier models differed by roughly 50 percentage points in overall error detection across both corpora. Replacing generic prompts with structured instructions that detailed error categories narrowed this gap and triggered tool use in agentic systems, raising Claude Code’s detection rate by more than 30 percentage points.
- Specialized tooling is essential for literature and numeric verification: On astrophysics manuscripts, Skepthical achieved a 46.7% detection rate on citation errors—far outperforming unassisted models (9% to 27%)—by directly retrieving and reading cited PDFs. Claude Code achieved 91.1% detection on astrophysics numerics by executing ad-hoc Python validation scripts.
- Fine-tuned open-weights models failed the audit: The domain-tuned models OpenReviewer-8B and PaperAudit-Bench Deep detected fewer than 3% of injected errors across both subject areas, demonstrating that conversational training on past conference reviews does not equip models to locate technical errors.
- Mathematical checks generated high false-positive rates: While overall false-positive rates remained below 10% on clean copies for capable models, mathematical critique proved notably unreliable, accounting for half of all false alarms across models.
Why it matters for research practice
The Cambridge-led findings arrive at a pivotal moment for scholarly communication, as publishers establish formal publisher policies on AI in peer review to govern generative tools. The benchmark confirms that automated reviewers cannot yet replace expert human referees in holistic scientific evaluation, such as judging methodological novelty, experimental design, or conceptual importance. Instead, language models operate most reliably as specialized verification assistants targeting discrete, verifiable claims.
For conference program committees and journal editorial offices, the study cautions against deploying unguided LLM reviewers for triage. Without external sandboxes to execute code and verify literature, automated systems routinely overlook subtle errors and misattribute citations. Furthermore, because mathematical critique displays high false-alarm rates, automated flags cannot serve as unreviewed rejections without placing unfair burdens on authors.
For institutional oversight, the results reinforce standards established for evaluating AI detection tools for research integrity: algorithmic screening must incorporate human-in-the-loop adjudication and transparent confidence scoring. Multi-agent audit pipelines like Skepthical offer immediate utility when deployed as pre-submission screening tools for authors or as verification layers during post-publication peer review, helping integrity teams systematically verify citations, formulas, and calculations when evaluating preprints versus peer-reviewed manuscripts.
Sources
- Íñigo Zubeldia, Boris Bolliet, Francisco Villaescusa-Navarro, Pablo Villanueva-Domingo, Miles Cranmer, Auditing AI peer reviewers: a benchmark on real scientific papers, arXiv:2611.17052v1 (2026-11-12)
