Skip to main content
v2026.11,986 entries · CC-BY 4.0
CASRAIAsk CASRAI · AI assistant for researchAsk a research question. Answers come with their sources.Funder rules, compliance, methods and publishing. Answers link the sources they rest on.

Editorial · CASRAI · AI and ML research outputs

Natural Code Rewrites Suppress CodeQL Alerts While Leaving Flaws Intact

Semantics-preserving natural code rewrites silence CodeQL security alerts across 26.2% of tested flows while leaving the underlying vulnerabilities intact.

Published 15 Nov 2026· 4 minute read

Ask CASRAI · free to try

Ask about this story

2 free questions a day, no account.

An AI assistant for research: funder rules, compliance, methods and publishing. It links the sources behind its answers and labels answers that come from a web search.

Answers draw on CASRAI's guides and dictionary plus the federal and funder documents we index: Federal Register, Grants.gov, Regulations.gov and UKRI.

Works on this site and inside Claude, Cursor and the AI tools you already use.

Everything CASRAI publishes — this page, the dictionary, the guides and the news — stays open to read, with no account and no card.

CASRAI is the reference for research administration — bookmark it for the next question.

On November 12, 2026, computer science researchers Zhezhen Cao, Weiyuan Ding, and Bowen Xu at North Carolina State University published an empirical investigation on arXiv (arXiv:2611.17123v1) demonstrating that semantics-preserving, natural code rewrites can systematically suppress security alerts from GitHub’s CodeQL static analysis engine while leaving underlying vulnerabilities completely exploitable. Evaluating 172 real-world vulnerability flows across popular npm packages, the researchers found that ordinary-looking maintenance refactorings generated by a large language model silenced CodeQL taint tracking in 26.2% of cases, highlighting a structural vulnerability in how continuous integration security gates are interpreted.

The Flawed Assumption of Silent Fixes in Static Analysis

Static application security testing (SAST) tools such as CodeQL serve as primary security gates within continuous integration pipelines, automated dependency maintenance, and pull-request review workflows. When a static analyzer flags a taint flow—where untrusted user input reaches a sensitive execution sink—development teams and automated remediation agents routinely infer that remediating the warning resolves the risk. Crucially, the researchers note that security gates are frequently read in reverse: when a previously reported alert disappears following a commit, developers assume the vulnerability has been remediated.

To test whether disappearing alerts represent genuine remediation or fragile analysis, the authors subjected CodeQL CLI (version 2.22.4, verified against version 2.27.3) to automated, vulnerability-preserving natural rewrites. Using Claude Opus 4.7 with zero knowledge of CodeQL’s internal implementation—provided only with the public alert location and sink description—the authors generated three plausible refactoring attempts per vulnerable flow. Each candidate rewrite underwent manual auditing and controlled dynamic execution to verify that the security flaw remained actively exploitable.

Empirical Results Across Vulnerability Classes

Across 172 taint flows extracted from the VulcaN benchmark of historical npm advisories, three natural rewrites per flow achieved an overall silencing rate of 26.2% (pass@3), silencing 45 distinct vulnerability flows at an average generation cost of just $0.51 per silenced flow. Crucially, in a control baseline where the same model refactored the identical functions without visibility into the security alert, 0.0% of the 516 attempts silenced CodeQL, confirming that alert suppression is driven specifically by alert-focused refactoring rather than generic code modernization.

Vulnerability Class Flows Attempts Silencing Rewrites pass@1 (%) pass@3 (%) 95% Confidence Interval
CWE-22 (Path Traversal) 91 272 47 17.2 33.0 [24.2, 43.1]
CWE-23 (Relative Path Traversal) 14 42 6 14.3 21.4 [7.6, 47.6]
CWE-77 (Command Injection) 11 32 5 15.2 36.4 [15.2, 64.6]
CWE-78 (OS Command Injection) 49 147 11 7.5 14.3 [7.1, 26.7]
CWE-79 (Cross-Site Scripting) 7 21 1 4.8 14.3 [2.6, 51.3]
Overall Evaluated Flows 172 514 70 13.6 26.2 [20.2, 33.2]
Benign Control Baseline 172 516 0 0.0 0.0 [0.0, 2.2]

Seven Gaps at the Library Modeling Boundary

Through differential ablation testing, the researchers traced every one of the 70 confirmed silencing rewrites to one of seven precise gaps in how static analyzers model runtime libraries. Rather than relying on obscure APIs, 64 of the 70 silencing rewrites (91.4%) exploited narrow modeling gaps inside standard APIs that CodeQL already explicitly models:

  • WHATWG URL Constructor (43 cases): CodeQL models the legacy Node.js url.parse API, but when code is updated to standard new URL(...), CodeQL propagates taint exclusively to query components while failing to track taint into .pathname or .href.
  • Unmodeled String Join Separators (10 cases): Taint tracking breaks when array elements are combined using separators that query specifications omit, including path separators (path.sep), slashes, or shell operators.
  • Non-Literal Array Receivers (7 cases): CodeQL drops taint during .join() operations whenever the receiver array is built dynamically via push, concat, or passed through helper parameters rather than defined as an inline array literal.
  • Function Dispatch and Format Variations (10 cases): Additional gaps involved built-ins invoked via .apply, unmodeled calls to path.format() despite full modeling of path.join(), and computed dispatch table lookups.

Cross-Analyzer Transfer and Industry Implications

The vulnerability-silencing phenomenon is not isolated to CodeQL. When the silencing rewrites were evaluated against two independent Node.js security analyzers—the graph-based semantic taint engine ODGen and the pattern scanner njsscan—the omissions readily transferred. Over 74% of the rewrites that silenced CodeQL also blinded ODGen to the same vulnerabilities, and 53% escaped all three analyzers combined. This cross-tool transfer proves that static analysis modeling boundaries represent widespread blind spots across the software ecosystem.

These findings carry urgent implications for code review for research software and academic computing environments. In research software engineering, laboratories increasingly rely on automated CI/CD checks to certify that shared computational tools and web services meet baseline compliance. If an automated remediation tool or refactoring commit clears a static analysis alert without addressing the dataflow path, project maintainers face an acute risk of false assurance. When evaluating automated code repair benchmarks and conducting a reproducible AI experiment, benchmark creators must not rely on disappearing static warnings as a proxy for correctness.

Institutions managing critical open-source research infrastructure under FAIR4RS software citation principles and conference evaluation frameworks such as ACM artifact review and badging should require end-to-end exploit regression tests and rigorous peer review rather than trusting passive static analysis alone.

Sources

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →