Skip to main content
v2026.11,986 entries · CC-BY 4.0

Editorial · CASRAI · AI and ML research outputs

Counterfactual Study Finds RAG Citations Often Reflect Parametric Memory

A 2026 study shows RAG citations frequently reflect parametric memory rather than retrieved sources, bypassing standard natural language inference checks.

Published 15 Nov 2026· 4 minute read

A study released on November 12, 2026, reveals that retrieval-augmented generation (RAG) systems frequently attach citations to retrieved literature that played no causal role in generating the cited statement. Researchers Tanzeel Abbas and Tehseen Zia of COMSATS University Islamabad, alongside David Windridge of Middlesex University London, demonstrated through counterfactual intervention that between 1.2% and 15.7% of citable statements produced by popular open-weight models originate in internal parametric memory rather than the supplied contextual sources, creating an illusion of empirical grounding that evades standard automated verification.

Measuring Provenance Through Counterfactual Substitution

Automated literature search and discovery platforms increasingly rely on RAG architectures to ground generative summaries in peer-reviewed passages. While existing evaluation frameworks assess citation correctness by measuring whether a retrieved snippet semantically entails an output claim, they cannot determine causal provenance—whether the retrieved snippet actually produced the claim. To isolate true origin, the authors introduced counterfactual value substitution: replacing numerical facts within retrieved passages and their titles with false variants before prompting the model.

When an artificial intelligence model generates an answer reflecting the substituted variant, its output causally depends on contextual retrieval. Conversely, if the model repeats the original factual value despite its complete absence from the retrieved documents, the claim derives strictly from memorized model weights. In developing the protocol, the authors identified a critical measurement artifact: failing to alter passage titles allowed models to extract original numbers from headings, which uncorrected workflows misclassified as parametric generation and overstated by nearly twofold.

Scale and Architecture: How Memory Overrides Context

Evaluating five parameter sizes of Qwen2.5-Instruct (0.5B to 14B) and Llama-3.1-8B-Instruct on the ASQA benchmark using GTR retrieval, the researchers observed that parametric provenance rises systematically with parameter scale. Within the Qwen family, parametric claim rates climbed monotonically from 1.2% in the 0.5B model to 13.3% in the 14B model (Kendall’s τ = 0.80). Model family architecture also exerted substantial influence: Llama-3.1-8B exhibited a 15.7% parametric rate, nearly double that of the similarly sized Qwen2.5-7B (8.2%).

Critically, models attached citations to parametric statements at almost identical frequencies as genuine contextual extractions. Across tested configurations, models cited 89% to 100% of their parametric claims, demonstrating that modern architectures routinely generate assertions from memory and retrospectively attach corroborating external citations.

Model Parameters Parametric Rate (95% CI) Parametric Claims Cited Contextual Claims Cited
Qwen2.5-0.5B 0.5B 1.2% [0.6%, 2.6%] 17% (1/6) 12% (58/491)
Qwen2.5-1.5B 1.5B 6.5% [4.5%, 9.3%] 44% (12/27) 53% (206/388)
Qwen2.5-3B 3.0B 8.3% [6.1%, 11.2%] 95% (36/38) 94% (392/417)
Qwen2.5-7B 7.6B 8.2% [6.0%, 11.2%] 100% (35/35) 96% (373/390)
Qwen2.5-14B 14.8B 13.3% [10.4%, 16.9%] 89% (50/56) 93% (209/224)
Llama-3.1-8B 8.0B 15.7% [12.6%, 19.4%] 94% (66/70) 98% (362/371)

Why Entailment Verification Fails to Detect Post-Rationalization

The findings expose a fundamental blind spot in standard verification pipelines that rely on Natural Language Inference (NLI). Evaluating a DeBERTa NLI verifier across all retrieved passages yielded a mean ROC AUC of 0.514, indistinguishable from random chance. Even when constraining the entailment check specifically to the cited passage, the detector achieved an AUROC of only 0.532 on Qwen2.5-7B.

Because post-rationalizing models select references that agree with their internally generated assertions, the resulting citations satisfy semantic entailment despite misrepresenting factual provenance. Comparing ten attribution methods, the authors found that contrastive probability distributions—measuring token-level entropy shifts before and after context arrival—separated provenance best (AUROC ~0.79). However, because these signals primarily detect whether a model possessed prior knowledge rather than whether it actively deployed it during decoding, automated filtering cannot reliably flag parametric leakage without incurring prohibitive false-alarm rates.

Implications for Evidence Synthesis and Research Integrity

These findings present direct operational challenges for academic workflows and scholarly infrastructure. Automated research assistance tools, such as Petal AI, and academic literature pipelines rely heavily on retrieval augmentation to construct reliable evidence summaries. In institutional research offices and synthesis teams, maintaining strict distinction between a formal citation vs reference is critical: a citation signifies verifiable evidential origin, not merely a topically compatible text.

For research teams conducting a thematic literature review or preparing a formal best evidence synthesis, unverified RAG citations introduce subtle contamination. When a generative system outputs a pre-existing bias or unverified training memory and anchors it with a post-hoc reference, peer reviewers cannot discern provenance through surface-level reading. Preserving research governance within institutional integrity and compliance frameworks will require audit mechanisms capable of tracing causal data flow rather than trusting surface-level textual agreement.

Sources

  • Abbas, T., Zia, T., and Windridge, D. (2026). “Cited but Unused: Parametric Provenance in Retrieval-Augmented Generation.” arXiv:2611.17161v1. https://arxiv.org/abs/2611.17161 (Checked 2026-11-15).
  • Abbas, T., Zia, T., and Windridge, D. (2026). “Cited but Unused: Parametric Provenance in Retrieval-Augmented Generation (HTML Full Text).” arXiv:2611.17161v1. https://arxiv.org/html/2611.17161v1 (Checked 2026-11-15).

Ask CASRAI · free to try

Ask about this story

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →