On November 8, 2026, researchers at RWTH Aachen University and Forschungszentrum Jülich posted a preprint on arXiv titled Building and Auditing Chemical Datasets with LLMs: A General LLM-as-a-Judge Framework Applied to Homogeneous Catalytic PET Depolymerisation. Led by Muhammad bin Javaid, Karim K. Ben Hicham, Enkhnyam Battulga, Alexander Mitsos, and Martin Grohe, the study introduces an automated verification architecture that audits literature-mined experimental records without ground-truth labels, establishing an auditable provenance framework for chemical research data curation.
What happened
Automated literature extraction using large language models has accelerated the compilation of structured scientific databases, yet verifying thousands of extracted multivariate records remains a critical barrier when pre-existing ground-truth benchmarks do not exist. In chemistry, experimental conditions are rarely reported in uniform tables; authors frequently describe baseline reagent quantities in procedural text and report variations across fragmented tables or footnotes, leading extraction models to conflate runs or miscalculate absolute masses.
To overcome this limitation, the Aachen research team designed a two-phase, citation-grounded pipeline applied to the literature on homogeneous catalytic poly(ethylene terephthalate) (PET) depolymerisation. The workflow separates primary data extraction from quality auditing:
- Phase 1 (Primary Extraction): A primary extractor model (GPT-5.6 Luna) processes document chunks parsed by Docling and enforces strict output structures via LiteLLM according to a standardized metadata schema. The prompt requires the extractor to assign universally unique identifiers (UUIDs) of referenced source chunks to each record, anchoring every extracted reaction condition in specific text or table paragraphs.
- Phase 2 (LLM-as-a-Judge Auditing): An independent open-weights auditor model (GPT-OSS-120B), hosted on an institutional endpoint with an unmetered 131,072-token context window, evaluates extracted JSON records directly against the complete chunked paper text. The judge produces step-by-step reasoning citing chunk UUIDs, checks chemical synonymy and mathematical consistency, issues a verdict, and proposes specific cited corrections for unsupported fields.
Methodology and key findings
Before corpus-wide deployment, the authors benchmarked the auditing pipeline against a hand-curated reference dataset of 295 ionic-liquid-catalysed PET glycolysis experiments spanning 24 papers. Prompt ablation experiments showed that providing a single worked example yielded optimal formatting guidance, raising extraction F1 score by +0.025 over zero-shot extraction while saving tens of thousands of prompt tokens compared to multi-shot prompts. Requiring explicit chunk-level UUID citations improved extraction F1 score by +0.064 (from 0.716 to 0.780), demonstrating that citation grounding serves as an attention anchor that actively improves extraction fidelity.
In validation against human evaluation across 112 stratified sample records, manual review agreed with the LLM judge’s verdict on 85.7% of records (96 of 112). All 49 records accepted by both the LLM judge and a heuristic bipartite-matching grader were confirmed correct under human review. Across 120 reviewed judge field-level interventions, 49% were demonstrably correct fixes and 48% represented valid chemical synonymy or benign reporting conventions; only 3% (4 corrections) were erroneous, indicating that 97% of judge interventions were either accurate corrections or harmless variants. In head-to-head comparisons on discordant records where only one automated grader agreed with human experts, the LLM judge outperformed the heuristic baseline 43 to 8 (p = 7 × 10⁻⁷ by McNemar’s test).
Deployed across an operational corpus mined from 561 papers, the system extracted 3,942 homogeneous catalytic PET depolymerisation experimental records. The LLM judge audited the complete dataset, accepting 67.5% (2,661 records) as written, rejecting 30.5% (1,202 records) with proposed cited corrections, and rejecting 2.0% (79 records) outright. Proposed corrections concentrated heavily in quantitative mass fields—specifically catalyst mass (508 corrections) and solvent mass (327 corrections)—where extractors struggle to carry context from narrative methods into result tables.
Why it matters for research practice
The preprint addresses an urgent challenge in open science and machine learning: how to conduct a trustworthy reproducibility audit on large-scale literature-mined databases when manual curation is cost-prohibitive. Traditional heuristic filters rely on brittle string matching that penalizes valid chemical synonyms and fails on complex multi-run papers. By pairing an independent open-weights judge with chunk-level citations, research teams can triage human curation, directing domain experts exclusively toward conflicting or flagged records.
Furthermore, the workflow strengthens compliance with FAIR data principles by establishing end-to-end data provenance. Rather than publishing unverified model outputs, the authors released the full database of 3,942 records alongside comprehensive judge audit logs, changelogs, and an open-source web application, the Chemistry Data Extractor Toolkit. This local-first tool pairs PDF viewing with synchronized UUID chunk highlighting, allowing curators to inspect model reasoning and verify extracted reaction parameters with a single click.
Sources
- Javaid, M. b., Ben Hicham, K. K., Battulga, E., Mitsos, A., and Grohe, M. (2026). Building and Auditing Chemical Datasets with LLMs: A General LLM-as-a-Judge Framework Applied to Homogeneous Catalytic PET Depolymerisation. arXiv preprint arXiv:2611.11030v1. https://arxiv.org/abs/2611.11030
- PET Depolymerisation Database and Evaluation Benchmark Repository. GitHub. https://github.com/Enkhnyam/pet-depolymerisation-database
- Chemistry Data Extractor Toolkit. GitHub. https://github.com/Enkhnyam/chemistry-data-extractor-toolkit
