Is AI Red-Teaming Security Theater? What the Research Says

CASRAI’s AI Red Teaming: Definition and How It Works in Frontier AI Safety covers what the term means inside a frontier developer’s own safety framework — the RSP, Preparedness Framework, and FSF family of commitments — and how that differs from generic application-security red-teaming. That page is deliberately definitional; it does not ask whether red-teaming, as currently practiced, actually delivers what its use in policy and corporate messaging implies. This page asks that second question, through the lens of a 2024 peer-reviewed study that put the practice under direct empirical scrutiny.

The paper: a CMU team surveyed the field and analyzed six real exercises

“Red-Teaming for Generative AI: Silver Bullet or Security Theater?” is a peer-reviewed paper by Michael Feffer, Anusha Sinha, Wesley Hanwen Deng, Zachary C. Lipton, and Hoda Heidari, all of Carnegie Mellon University, published at AIES 2024 (the AAAI/ACM Conference on AI, Ethics, and Society). It first appeared on arXiv on January 29, 2024 and was last revised August 27, 2024 (v3, arXiv:2401.15897). The authors did two things: they ran a structured literature survey — searching arXiv, Google Scholar, OpenReview, the ACL Anthology, and the ACM Digital Library for “red-teaming,” “ai red-teaming,” “jailbreak,” and “llm jailbreak,” supplemented by snowball sampling — and they analyzed six documented real-world red-teaming exercises as case studies: Microsoft’s work on Bing Chat, OpenAI’s on GPT-4, DeepMind’s on Gopher, Anthropic’s on both Claude 1 and Claude 2, and a DEF CON AI Village public red-teaming event. They also reviewed public comments submitted to a NIST Request for Information on red-teaming generative AI.

What “security theater” actually claims — and what it doesn’t

The paper’s own words matter here, because the phrase is easy to overread as “red-teaming doesn’t work.” That is not the claim. Quoting the abstract directly: the authors argue “that while red-teaming may be a valuable big-tent idea for characterizing GenAI harm mitigations, and that industry may effectively apply red-teaming and other strategies behind closed doors to safeguard AI, gestures towards red-teaming (based on public definitions) as a panacea for every possible risk verge on security theater.” The critique targets the gap between what red-teaming is publicly represented as capable of — a general-purpose safety guarantee, invoked to answer nearly any regulatory concern — and what a specific, time-boxed evaluation of a specific artifact against a specific threat model can actually establish. Their own case studies, discussed below, found real harmful behavior in every single exercise; the paper’s argument is about the practice’s public framing and its lack of structure, not about whether red-teaming finds problems when it’s done.

Where practitioner definitions diverge

The paper’s central empirical finding is that AI red-teaming, “as defined in public literature,” lacks consensus on scope, structure, and assessment criteria. The authors identify four axes along which practice diverges: the threat model the activity probes (if one is specified at all), the artifact under evaluation, the setting in which the activity is conducted (actors, resources, methodology, test-beds), and the decisions the activity instigates afterward (reporting, disclosure, mitigation).

The paper illustrates the definitional problem with the source that gave red-teaming its biggest policy platform: the 2023 US presidential Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, which mentions red-teaming eight times and defines it as “a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with developers of AI… most often performed by dedicated ‘red teams’ that adopt adversarial methods to identify flaws and vulnerabilities, such as harmful or discriminatory outputs from an AI system, unforeseen or undesirable system behaviors, limitations, or potential risks associated with the misuse of the system.” The paper argues that definition leaves the operative questions unanswered: what risks can or should be caught this way, how the activity should be structured, who besides the developer should be at the table, and how findings should be documented and reported. The Frontier Model Forum’s own words, quoted in the paper, make the same point from industry’s side: “there is currently a lack of clarity on how to define ‘AI red teaming’ and what approaches are considered part of the expanded role it plays in the AI development life cycle.”

What six real exercises actually showed

The paper’s case-study table lists the exercises analyzed: Bing Chat (Microsoft), GPT-4 (OpenAI), Gopher (DeepMind), Claude 2 (Anthropic), a DEF CON AI Village event (various developers), and Claude 1 (Anthropic) — all sourced from public reports and news coverage, since, as the authors note, “industry teams do not disclose (all of) their methods.” Three findings stand out:

• Team composition varied by type, not just by size. The authors group the evaluator teams into three categories: handpicked subject-matter experts (internal and external), crowdsourced participants drawn from crowdworking platforms or live events, and language models prompted or fine-tuned to red-team other models. Each type produced different resource profiles and different blind spots — crowdsourced evaluators, time-boxed to roughly 30–50 minutes per task, tended to repeatedly probe the same easy-to-trigger risk areas rather than explore the full space, while subject-matter-expert teams on two of the six cases red-teamed continuously for 6–7 months before release.
• No standard for what gets disclosed. Specific examples of harmful model behavior were publicly shared in only half of the six cases. One case released a full dataset of 38,961 red-team attacks publicly; two others shared example outputs without releasing the full scope of attacks; national-security-sensitive findings weren’t shared at all pending a still-under-development responsible-disclosure process. Cost figures (compute, time, money) were disclosed mainly for crowdsourced teams — e.g., the hourly rate paid to crowdworkers — and largely absent for expert and model-driven teams.
• No standard for what happens after. Every one of the six exercises found problematic behavior; none resulted in a decision not to release the model. Mitigation strategies ranged from concrete (adversarial fine-tuning) to vague, and the paper warns this pattern “could reduce red-teaming to an approval-stamping process wherein one can say that red-teaming was performed as an assurance without providing further details into issues discovered or fixed.”

Industry’s own comments to NIST said the same thing

The paper’s fifth section analyzes public comments submitted to a NIST Request for Information on red-teaming generative AI, issued under the same 2023 Executive Order. The authors report that “industry, academia, and civil society organizations suggest that NIST should specify a clearer definition of ‘red-teaming'” and that “even industry firms with experience red-teaming GenAI expressed a desire for concrete guidance from NIST” — external corroboration, from the people doing the work, of the paper’s own literature-and-case-study finding that the practice is “loosely structured and perhaps not the rigorous practice implied by the Executive Order.”

The proposed fix: a rigor toolkit, not retirement

The paper’s stated recommendation is explicit that red-teaming should continue, alongside other evaluation methods, not be abandoned: “red-teaming should therefore be considered as one evaluation paradigm, among others such as algorithmic impact assessments and audits.” Its concrete contribution is a question bank (the paper’s Table 1) organized into three phases, meant to scaffold future red-teaming activities rather than serve as a finished standard:

• Phase 0, pre-activity: What artifact is under evaluation, and what version/fine-tuning state? What threat model does the activity probe — illustrative, broad, or a specific named vulnerability? What are the criteria for calling the activity a success, and can it be reproduced? Who is on the red team, what is the internal/external and demographic mix, and what are their incentives to participate?
• Phase 1, during the activity: What resources (time, compute) do participants have, and do those resources realistically mirror an adversary’s? What access do participants have to the model, and what methods and auxiliary tools — including AI tools — can they use?
• Phase 2, post-activity: What reports are produced, who gets access to them and when, and what justification is given for anything withheld or delayed? What resources did the activity actually consume? What mitigations are proposed, who is accountable for implementing them, and how will their effectiveness be evaluated?

The authors are explicit that the question bank is a starting point, not a finished standard: “we emphasize that these are not finalized guidelines but rather (what we hope is) the start of a broader conversation about GenAI red-teaming and evaluation processes,” and they name co-design, evaluation, and development of scoring rubrics as open future work.

The NIKOLAI angle: red-teaming as an unstructured instance of “external review”

CASRAI’s own NIKOLAI project — an independent, unendorsed reference dictionary of frontier-AI-safety terms, not a standard the CMU authors or anyone else has adopted — carries a track, N8 (“Transparency and Review”), built around exactly the gap this paper describes: getting an assessment’s type, scope, and output onto the record. The track’s closest element to red-teaming, verified live before drafting this page, is the External Review element within N8, defined as “a record of an assessment performed by a party outside the model developer — of a model, a risk report, a safeguard, or compliance with a framework — capturing the review’s type, scope, and output.” When red-teaming is performed by outside participants — as in four of the paper’s six case studies, including Anthropic’s own RSP commitment to “partner with a diverse range of external red team and penetration testing experts” — it is a specific instance of what the External Review element is defined to capture.

Worth stating plainly: none of the eleven rows in the External Review element’s existing crosswalk table — which maps Anthropic (twice, for its Risk Report and its Policy on AI Exponential), OpenAI, Google DeepMind, the EU, California SB 53, NIST CAISI/UK AISI, METR, the Frontier Model Forum, G42, and the FRONTIER Act against the element’s definition — currently names red-teaming specifically. It appears only once on the page, in a separate “related, not mapped” note pointing at Meta’s Advanced AI Scaling Framework v2, which uses red teaming “when appropriate” alongside external experts in threat-modeling workshops. That gap is itself consistent with the CMU paper’s core finding: the same terminological fragmentation that keeps “red-teaming” from meaning one thing across labs also means it hasn’t yet been mapped as its own row against a transparency element built to record exactly what the paper says red-teaming reports currently omit — type, scope, and output. Every row on that element, and this observation about it, is CASRAI’s own reading; NIKOLAI’s shadow-mapping disclaimer applies here exactly as it does everywhere else on the site: “No lab, evaluator or regulator named on a shadow row has declared, endorsed, or been consulted on it.” The CMU authors did not write with NIKOLAI in mind and have not endorsed this framing.

How this page differs from CASRAI’s other red-teaming coverage

To be explicit about scope: CASRAI’s AI Red Teaming Explained guide and the Red-teaming dictionary term answer “what is it and what does it mean inside a safety framework.” This page answers a different question — “does the practice, as currently documented and disclosed, hold up to scrutiny” — using one specific, citable piece of peer-reviewed research as its evidence base rather than framework language alone. Read together with CASRAI’s guides to Responsible Scaling Policy commitments, evaluator independence, and third-party evaluator standards, the picture is consistent: red-teaming is a real and useful practice that, per this paper’s own evidence, currently lacks the shared definitions and reporting standards that would let an outside reader compare one lab’s red-teaming claim against another’s.

Last verified September 20, 2026. “Red-Teaming for Generative AI: Silver Bullet or Security Theater?” (Feffer, Sinha, Deng, Lipton, Heidari; Carnegie Mellon University; AIES 2024) quoted directly from the full text of arXiv:2401.15897v3 (last revised August 27, 2024), retrieved and extracted from the arXiv PDF. Executive Order and Frontier Model Forum language quoted as reproduced verbatim in the paper’s own text. NIKOLAI’s N8 track and its External Review element (definition, disclaimer, and full 11-row crosswalk) read directly from the live /nikolai/element/external-review/ page before drafting; no crosswalk row currently names red-teaming, and this page does not claim one exists. Dedup-checked against live WordPress (exact-slug and fuzzy search across guide, comparison, and news) immediately before publication — CASRAI’s existing AI Red Teaming Explained guide (WP 33539) is purely definitional and contains no methodology critique of red-teaming rigor, confirming this page covers new ground rather than duplicating it.