AI Red Teaming: Definition and How It Works in Frontier AI Safety

Most search results for “AI red teaming” describe a commercial application-security service: a vendor adversarially tests a deployed LLM product for prompt injection, jailbreaks, and data leakage, the same way a firm would penetration-test any web application. That is a real and useful practice, and CASRAI’s dictionary defines it on its own terms — see Red-teaming for the general adversarial-testing concept. This guide is about a narrower, distinct use of the same term: red-teaming as a published safeguard-testing commitment inside a frontier AI developer’s own safety framework — the Responsible Scaling Policy (RSP), Preparedness Framework, or Frontier Safety Framework (FSF) family of documents covered in CASRAI’s guide to how a Responsible Scaling Policy works. The two uses overlap in method but answer different questions, and conflating them means missing what a frontier lab is actually committing to when its safety framework mentions red-teaming.

What “AI red teaming” means inside a frontier safety framework

Anthropic’s Responsible Scaling Policy is the clearest published example of red-teaming appearing as a named commitment rather than a general practice. Under its planned ASL-3 security safeguards, the RSP commits Anthropic to “partner with a diverse range of external red team and penetration testing experts” and to “simulate sophisticated attacks, including insider threat and software supply chain compromise scenarios, to identify vulnerabilities” in the controls protecting a frontier model’s weights. Separately, under deployment safeguards, the RSP describes its real-time misuse classifiers as kept current using data from “our asynchronous monitoring system and insights from our incident response protocol, bug bounty program, and internal and external red-teaming efforts.” In both places, red-teaming is doing the same job: trying to break a specific, already-built safeguard, not testing what the underlying model can do.

The two things frontier-lab red-teaming actually tests

Read closely, Anthropic’s RSP text points to two distinct red-teaming targets, and the distinction matters for what a disclosure actually tells you:

• Model weight security. External red-team and penetration-testing partners try to defeat the technical and procedural controls guarding a frontier model’s trained weights — the premise being that if the weights leak or are stolen, every downstream safety measure built into the deployed product can be bypassed by simply running the model without them. This is security red-teaming in the classic information-security sense, aimed at the lab’s infrastructure and access controls rather than at the model’s own outputs.
• Deployment-safeguard resistance. Internal and external red-teaming efforts also probe the safeguards that sit between a deployed model and a user — the classifiers and filters meant to catch harmful or policy-violating requests — to find jailbreaks and obfuscation techniques those safeguards miss. This is closer to what most AppSec red-teaming content on the web describes, but here it feeds a specific, named safeguard tier under a published policy rather than a one-off vendor engagement.

A government evaluator does comparable work from outside the lab. CASRAI’s guide to CAISI and the UK AI Security Institute reports that the UK AI Security Institute’s “safeguard-testing work has also produced automated jailbreak techniques capable of bypassing the defenses of well-protected systems, developed as part of its own red-teaming rather than reported by an outside party” — red-teaming run by a third-party government body, independent of the lab whose safeguards it is testing.

How this differs from a dangerous-capability evaluation

It is easy to conflate red-teaming with the dangerous-capability evaluations that drive whether a model has crossed a capability threshold under an RSP, a Critical Capability Level under an FSF, or a tracked-category threshold under a Preparedness Framework. They are not the same practice, and CASRAI’s guide to how a Responsible Scaling Policy works covers the capability-evaluation side in depth: those evaluations are domain-specific tests, combining automated benchmarks with structured human-uplift studies and careful capability elicitation, asking whether a model can do something — provide meaningful counterfactual assistance toward a bioweapon, automate a cyberattack end to end, meaningfully accelerate AI research.

Red-teaming, in the frontier-safety-framework sense described above, asks a different question: once a threshold is crossed and a safeguard is required, can that safeguard actually be defeated? A model can fail every dangerous-capability evaluation and still be a red-teaming target, because the weight-security controls protecting it are tested regardless of what the model itself can do. Conversely, a model that trips a capability threshold gets new red-teaming obligations as part of the stronger safeguard tier that threshold requires — the two practices are sequenced, not interchangeable: capability evaluation decides whether stronger safeguards are required; red-teaming checks whether the safeguards that were promised actually hold.

How this differs from generic AppSec red-teaming

The audience searching “AI red teaming” is overwhelmingly looking for the commercial AppSec service: a vendor is engaged to probe one deployed LLM application for prompt injection, jailbreaks, and sensitive-data leakage, scoped to that one product, typically as a paid, discrete engagement. That is a legitimate and separate field — again, see CASRAI’s dictionary entry on red-teaming for that general definition, which also covers the human-versus-automated distinction in more depth than is relevant here.

Frontier-lab red-teaming, as described in this guide, differs in three ways: it is scoped to a specific model against a specific published safety framework rather than to one deployed application; it is tied to a public accountability commitment (a lab that describes red-teaming in its RSP, Preparedness Framework, or FSF is stating what it does as a matter of policy, not simply as a security practice it happens to follow); and its findings are meant to feed a specific downstream decision — whether a required safeguard tier is actually in place — rather than a general security report to whoever commissioned the test.

What a red-teaming disclosure does and doesn’t tell you

A framework that says a lab “conducts red-teaming” is making a much thinner claim than it sounds like. None of the public RSP text quoted above specifies testing cadence, the number or seniority of external testers, what counts as a passed or failed red-team exercise, or whether findings are independently verified rather than self-reported. “We red-team our models” and “we red-team our models on a defined schedule, with named external partners, against a published pass bar” are very different claims, and current framework disclosures mostly stop well short of the second. This is exactly the gap NIKOLAI, CASRAI’s frontier-AI-safety dictionary, exists to make checkable: it crosswalks the vocabulary frontier-safety frameworks use — including capability threshold, safety case, and elicitation method — against what labs actually publish, element by element, rather than taking a framework’s own description of its practices at face value.

This guide can confirm red-teaming’s role directly from Anthropic’s own published RSP text, and from the UK AI Security Institute’s own account of its testing work as reported in CASRAI’s CAISI/AISI guide. OpenAI’s Preparedness Framework and Google DeepMind’s Frontier Safety Framework are widely understood to run comparable adversarial-testing programs, but their current published framework documents were not directly accessible while researching this guide, so their specific red-teaming commitments are not quoted here. See CASRAI’s guide to what counts as a frontier AI model and its explainer on AI alignment for related, independently verified context on why this category of model gets this category of scrutiny in the first place.

Frequently asked questions

Is AI red teaming the same as penetration testing?

They overlap but are not identical. Anthropic’s RSP explicitly pairs “external red team and penetration testing experts” as part of the same security-safeguard commitment. Penetration testing traditionally targets infrastructure — networks, servers, access controls. Red-teaming in this context covers that same infrastructure-security ground when it is defending model weights, and also extends to the model’s own behavior, such as whether a deployed system’s safeguards can be jailbroken.

Does red-teaming replace dangerous-capability evaluations?

No. They test different things. Capability evaluations, covered in CASRAI’s Responsible Scaling Policy guide, determine whether a model has crossed a defined dangerous-capability threshold. Red-teaming tests whether the safeguards required once a threshold is crossed — or the controls protecting the model’s weights regardless of threshold — can actually be defeated. A frontier safety framework requires both.

Who actually performs this kind of red-teaming?

Based on Anthropic’s published RSP, a mix of in-house teams and external partners: the policy commits to partnering with “a diverse range of external red team and penetration testing experts” alongside an in-house security team, and describes classifier-hardening work drawing on “internal and external red-teaming efforts.” Government evaluators also run independent red-teaming — CASRAI’s guide to CAISI and the UK AI Security Institute describes AISI producing its own automated jailbreak techniques as part of its safeguard-testing work.

If a lab says it “red-teams” its models, what does that actually confirm?

On its own, not much beyond the fact that some adversarial testing happened. It doesn’t tell you the cadence, the scope, who performed it, what would count as a failure, or whether results were independently checked rather than self-reported. Treat a bare red-teaming claim the way you would treat a bare safety-case claim: as something to check against what the framework itself requires, not as a completed verification.

Where can I find the generic, non-frontier-safety definition of AI red teaming?

CASRAI’s dictionary entry on red-teaming covers the broader adversarial-testing concept, including how human and automated red-teaming are typically combined, without the frontier-safety-framework scoping this guide adds.