On November 12, 2026, computer science researchers Zhezhen Cao, Weiyuan Ding, and Bowen Xu at North Carolina State University published an empirical investigation on arXiv (arXiv:2611.17123v1) demonstrating that semantics-preserving, natural code rewrites can systematically suppress security alerts from GitHub’s CodeQL static analysis engine while leaving underlying vulnerabilities completely exploitable. Evaluating 172 real-world vulnerability flows across popular npm packages, the researchers found that ordinary-looking maintenance refactorings generated by a large language model silenced CodeQL taint tracking in 26.2% of cases, highlighting a structural vulnerability in how continuous integration security gates are interpreted.
The Flawed Assumption of Silent Fixes in Static Analysis
Static application security testing (SAST) tools such as CodeQL serve as primary security gates within continuous integration pipelines, automated dependency maintenance, and pull-request review workflows. When a static analyzer flags a taint flow—where untrusted user input reaches a sensitive execution sink—development teams and automated remediation agents routinely infer that remediating the warning resolves the risk. Crucially, the researchers note that security gates are frequently read in reverse: when a previously reported alert disappears following a commit, developers assume the vulnerability has been remediated.
To test whether disappearing alerts represent genuine remediation or fragile analysis, the authors subjected CodeQL CLI (version 2.22.4, verified against version 2.27.3) to automated, vulnerability-preserving natural rewrites. Using Claude Opus 4.7 with zero knowledge of CodeQL’s internal implementation—provided only with the public alert location and sink description—the authors generated three plausible refactoring attempts per vulnerable flow. Each candidate rewrite underwent manual auditing and controlled dynamic execution to verify that the security flaw remained actively exploitable.
Empirical Results Across Vulnerability Classes
Across 172 taint flows extracted from the VulcaN benchmark of historical npm advisories, three natural rewrites per flow achieved an overall silencing rate of 26.2% (pass@3), silencing 45 distinct vulnerability flows at an average generation cost of just $0.51 per silenced flow. Crucially, in a control baseline where the same model refactored the identical functions without visibility into the security alert, 0.0% of the 516 attempts silenced CodeQL, confirming that alert suppression is driven specifically by alert-focused refactoring rather than generic code modernization.
| Vulnerability Class | Flows | Attempts | Silencing Rewrites | pass@1 (%) | pass@3 (%) | 95% Confidence Interval |
|---|---|---|---|---|---|---|
| CWE-22 (Path Traversal) | 91 | 272 | 47 | 17.2 | 33.0 | [24.2, 43.1] |
| CWE-23 (Relative Path Traversal) | 14 | 42 | 6 | 14.3 | 21.4 | [7.6, 47.6] |
| CWE-77 (Command Injection) | 11 | 32 | 5 | 15.2 | 36.4 | [15.2, 64.6] |
| CWE-78 (OS Command Injection) | 49 | 147 | 11 | 7.5 | 14.3 | [7.1, 26.7] |
| CWE-79 (Cross-Site Scripting) | 7 | 21 | 1 | 4.8 | 14.3 | [2.6, 51.3] |
| Overall Evaluated Flows | 172 | 514 | 70 | 13.6 | 26.2 | [20.2, 33.2] |
| Benign Control Baseline | 172 | 516 | 0 | 0.0 | 0.0 | [0.0, 2.2] |
Seven Gaps at the Library Modeling Boundary
Through differential ablation testing, the researchers traced every one of the 70 confirmed silencing rewrites to one of seven precise gaps in how static analyzers model runtime libraries. Rather than relying on obscure APIs, 64 of the 70 silencing rewrites (91.4%) exploited narrow modeling gaps inside standard APIs that CodeQL already explicitly models:
- WHATWG URL Constructor (43 cases): CodeQL models the legacy Node.js
url.parseAPI, but when code is updated to standardnew URL(...), CodeQL propagates taint exclusively to query components while failing to track taint into.pathnameor.href. - Unmodeled String Join Separators (10 cases): Taint tracking breaks when array elements are combined using separators that query specifications omit, including path separators (
path.sep), slashes, or shell operators. - Non-Literal Array Receivers (7 cases): CodeQL drops taint during
.join()operations whenever the receiver array is built dynamically viapush,concat, or passed through helper parameters rather than defined as an inline array literal. - Function Dispatch and Format Variations (10 cases): Additional gaps involved built-ins invoked via
.apply, unmodeled calls topath.format()despite full modeling ofpath.join(), and computed dispatch table lookups.
Cross-Analyzer Transfer and Industry Implications
The vulnerability-silencing phenomenon is not isolated to CodeQL. When the silencing rewrites were evaluated against two independent Node.js security analyzers—the graph-based semantic taint engine ODGen and the pattern scanner njsscan—the omissions readily transferred. Over 74% of the rewrites that silenced CodeQL also blinded ODGen to the same vulnerabilities, and 53% escaped all three analyzers combined. This cross-tool transfer proves that static analysis modeling boundaries represent widespread blind spots across the software ecosystem.
These findings carry urgent implications for code review for research software and academic computing environments. In research software engineering, laboratories increasingly rely on automated CI/CD checks to certify that shared computational tools and web services meet baseline compliance. If an automated remediation tool or refactoring commit clears a static analysis alert without addressing the dataflow path, project maintainers face an acute risk of false assurance. When evaluating automated code repair benchmarks and conducting a reproducible AI experiment, benchmark creators must not rely on disappearing static warnings as a proxy for correctness.
Institutions managing critical open-source research infrastructure under FAIR4RS software citation principles and conference evaluation frameworks such as ACM artifact review and badging should require end-to-end exploit regression tests and rigorous peer review rather than trusting passive static analysis alone.
Sources
- North Carolina State University Research Preprint (arXiv:2611.17123v1) – https://arxiv.org/abs/2611.17123 (Checked 2026-11-15)
- VulcaN Vulnerability Benchmark for Node.js Applications – https://arxiv.org/html/2611.17123v1 (Checked 2026-11-15)







