Written and maintained by CASRAI Editorial Board
Last updated
Last verified against primary sources: 25 September 2026. Frontier AI evaluation is usually described as if it were one activity. It is not. A dangerous-capability evaluation asks what can this model do. A propensity or scheming evaluation asks what does this model tend to do when it thinks no one is watching. A control evaluation asks a third question that neither of those answers: if this model were actively working against the safeguards wrapped around it, would those safeguards still hold?
That third question is the one Redwood Research — a registered 501(c)(3) nonprofit working on threat assessment and mitigation for AI systems — turned into a repeatable experimental procedure. The organisation describes its own framing plainly: “An AI is controlled if it is unable to cause damage even if it is egregiously misaligned.” And the evaluation that follows from that framing measures, in Redwood’s words, “a model’s capability to subvert control techniques instead of its propensity to do so.”
The distinction between capability-to-subvert and propensity-to-subvert is the whole point, and it is why a control evaluation is a separate evidentiary category rather than a flavour of red teaming. This page walks the method, the numbers from the original paper, the assumption list it runs on, and the specific places where a governance document leaning on a control evaluation is leaning on something narrower than it looks.
Where Control Evaluations Sit Among the Other Evidence Types
It helps to place the method against the neighbours it is most often confused with.
| Method | Question it answers | What a good result licenses | Main failure mode |
|---|---|---|---|
| Dangerous-capability evaluation | Can the model do the harmful thing at all? | A capability upper bound, subject to elicitation quality | Under-elicitation — the model could, but you failed to get it to |
| Propensity / scheming evaluation | Does the model deceive, sandbag, or pursue hidden goals? | Evidence about behavioural tendencies in the tested conditions | The model behaves differently once it believes it is deployed |
| Red teaming (safeguard-focused) | Can a human attacker get the model to produce harmful output? | Evidence about jailbreak and misuse resistance | Unstructured effort; no defined stopping rule or denominator |
| Control evaluation | If the model itself were the adversary, would the deployment protocol still catch it? | A safety estimate for a protocol, conditional on stated assumptions | The red team was not as good as a real scheming model would be |
Two things in that last row are unusual. First, the object being evaluated is not the model — it is the protocol: the whole pipeline of monitoring, deferral, editing and human auditing that sits between the model and the world. Second, the result is explicitly conditional. A control evaluation does not claim the model is benign. It claims that even granting that the model is hostile and competent, the protocol holds to a stated degree.
This is the same move a security team makes when it stops arguing about whether an insider is trustworthy and starts asking what the access controls would survive. It is also why control evaluations produce a different kind of sentence in a safety document than Apollo Research’s scheming-detection work, which is a propensity method, or than METR’s autonomy and task-horizon evaluations, which are capability methods.
The Original Experiment: APPS Backdooring
The foundational paper is “AI Control: Improving Safety Despite Intentional Subversion” by Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan and Fabien Roger, first posted to arXiv on 12 December 2023 (arXiv:2312.06942; v5 dated 23 July 2024), and presented as an oral paper at ICML.
The setting is deliberately small enough to be run thousands of times. The blue team must get correct solutions to 5,000 interview-difficulty programming problems from the APPS dataset. The red team must get at least one backdoor through: code that passes every test case but produces a wrong answer on some input the red team knows and the blue team does not. GPT-4 is the untrusted model — more capable, assumed possibly hostile. GPT-3.5 is the trusted model — weaker, assumed not to be scheming. A limited budget of human auditing is available, standing in for the expensive, high-quality review a real deployment could afford on a small fraction of actions.
The protocols tested and their reported results:
| Protocol | Safety | Usefulness | What it does |
|---|---|---|---|
| Upfront auditing | 15% | 100% | Spend the entire audit budget on the first actions, then run unmonitored |
| Trusted monitoring | 62% | 97% | GPT-3.5 scores GPT-4’s code for suspiciousness; the most suspicious go to human audit |
| Untrusted monitoring | 79% | 96% | A separate GPT-4 instance monitors, prevented from coordinating with the policy instance |
| Trusted editing (20% threshold) | 92% | 94% | GPT-3.5 rewrites the suspicious fraction of GPT-4’s output rather than just flagging it |
Read the top row first, because it is the one that makes the method worth the trouble. Upfront auditing spends a real audit budget and still lands at 15% safety, because a patient adversary simply waits. The interesting finding is not that monitoring helps; it is that the ordering and structure of cheap oversight moves the number from 15% to 92% while costing six points of usefulness. That trade — safety against usefulness, at a fixed audit budget — is the output format of the method.
The Red Team Is the Measuring Instrument
The single most important thing to understand about a control evaluation is that the red team is not decoration. It is the instrument, and the reading is only as good as the instrument.
Because no current model is actually a competent schemer, the evaluation substitutes a human red team that builds an attack policy — a strategy the untrusted model is made to follow, designed to defeat the specific protocol under test. The safety number is then an estimate of how often the protocol catches the best attack the red team could construct. If a genuinely misaligned model would have found a better attack, the evaluation overstates safety by exactly that gap, and nothing inside the evaluation reveals the gap.
This is not a hidden weakness; the field states it openly. The UK AI Security Institute and Redwood co-authored “A sketch of an AI control safety case” (Tomek Korbak, Joshua Clymer, Benjamin Hilton, Buck Shlegeris and Geoffrey Irving, arXiv:2501.17315, 28 January 2025), which sets out what a developer would have to establish for a control evaluation to carry weight in a deployment argument. It rests on three claims:
- Elicitation sufficiency. The red team really did surface the model’s capability to pursue the harmful side task.
- Control persistence. The safety measures are at least as effective in deployment as they were in the evaluation.
- Conservative extrapolation. The measured results are projected to a deployment risk estimate in a way that errs toward caution.
Every one of those is an assumption doing load-bearing work, and only the third is under the developer’s full control. Claim one is the elicitation problem that haunts all frontier evaluation. Claim two is a statement about the production environment, not about the experiment. This is the same structural problem that makes safety cases hard to standardise across frontier AI governance: the arithmetic is easy, and the premises are where the argument lives.
The Conservative Assumptions, Named
Control evaluations buy their conditional guarantee by granting the adversary a great deal. Buck Shlegeris enumerated the standard grant list in “Thoughts on the conservative assumptions in AI control” (17 January 2025). The evaluation assumes the AIs:
- are consistently trying to subvert the safety measures;
- can strategise about subversion “totally illegibly (e.g. in a single forward pass)”;
- can follow a strategy coordinated between all their instances;
- know exactly what safety measures are in use;
- know a great deal about the distribution of inputs they will see;
- can pick Schelling points perfectly, for coordination without communication;
- can follow calibrated stochastic policies — attacking with probability 0.01%, say, and actually hitting that rate.
Shlegeris is candid that these are not uniformly conservative. Perfect Schelling-point selection ranges from implausible to plausible depending on capability level. Consistent subversion intent may not describe the first misaligned systems at all, which tend to be inconsistent rather than steadfast. He also notes the flip side that governance readers usually miss: if you allocate all your effort on the basis of classic control evaluations, you will underestimate the value of safety measures that work precisely because some of these assumptions are false. The assumption list is a floor, not a forecast.
Joshua Clymer, writing days after the safety-case sketch (“Takeaways from sketching a control safety case,” 30 January 2025), pushed the point further. He had expected control evaluations to work as a reliable black box; sketching one changed his mind, because “so many assumptions are made at every step.” Evaluations must be matched to a specific threat model — if the worry is code smuggled through JSON files, the test environment has to contain JSON-handling tasks. And the translation from experimental results to a risk claim is a judgement call, not a mechanical procedure.
From One Paper to Shared Infrastructure
The method stopped being a single lab’s paper somewhere around late 2025.
On 22 October 2025, the UK AI Security Institute — a research organisation within the Department for Science, Innovation and Technology — released ControlArena, a library for running control experiments, created jointly with Redwood Research and built on AISI’s Inspect evaluation platform. AISI stated at launch that it had already been used by researchers at Anthropic, Google DeepMind and Redwood Research among others. The public documentation now lists fifteen settings — AgentDojo, Apps, Bash, BashArena, BigCodeBench, Data Poisoning, Eval Sabotage, IAC, IAC Fast, Infra, Rogue Eval, SAE Interp Sabotage, SHADE Arena, SWE-bench Django and vLLM — each pairing a legitimate main task with a covert side task, plus implementations of common protocols such as defer-to-trusted.
That matters for governance for an unglamorous reason. A control evaluation result is only comparable across organisations if the setting, the audit budget and the attack-policy affordances are specified the same way. Shared infrastructure is what makes third-party evaluator methodology more than a description of one team’s taste.
The research has also started finding results that a governance reader needs to know about. “Evaluating Control Protocols for Untrusted AI Agents” (Jon Kutasov, Chloe Loughridge, Yuqi Sun, Henry Sleight, Buck Shlegeris, Tyler Tracy and Joe Benton, arXiv:2511.02997, 4 November 2025) tested protocols on the SHADE-Arena dataset and reported that resampling for incrimination and deferring on critical actions were the strongest, “increasing safety from 50% to 96%.” But when the attack policy was given knowledge of protocol internals — when it knew when resampling occurred, or could simulate the monitor — resampling collapsed to 17% safety. Deferring on critical actions stayed robust against the strongest red-team strategies tested.
The governance implication is sharp: some protocols derive their measured safety from secrecy about the protocol. A developer citing a control-evaluation number in a public safety framework should be able to say which kind of protocol it is, because a number that depends on the adversary’s ignorance decays as the adversary’s knowledge of the deployment grows — including through training data.
What the Method Does Not Establish
Five limits, stated plainly, because they are the ones that get elided when a control result is quoted in a policy document.
It does not establish that the model is aligned. A control evaluation is indifferent to alignment by construction. Using one to argue that a model is safe in itself inverts the method.
It does not cover non-scheming adversarial behaviour well. The classic framing assumes a consistent schemer. Models that are inconsistently misaligned, data-poisoned, or more easily jailbroken in deployment than in testing are not the modal case the method was built around, and extending control evaluations to those threat models is live research rather than settled practice.
It does not survive a threat-model mismatch. Clymer’s JSON example generalises: a control evaluation is evidence about the attack surface present in the setting, and silent about the surfaces that were not there.
It does not convert itself into a probability. The safety figure is a conditional estimate within a specified game, not a deployment risk rate. The extrapolation step is a human judgement that has to be shown, not asserted.
It is not, at present, a risk assessment at all. Ryan Greenblatt’s “An overview of areas of control work” (9 April 2025) is explicit: “Right now, models aren’t actually risky, so work which is directly focused on evaluating risk doesn’t provide that much of the value.” The current function of control evaluations is to develop and compare countermeasures, with about a third of effort recommended toward building the settings themselves. They are being built now so that they can be load-bearing later.
How This Lands in NIKOLAI
CASRAI maintains NIKOLAI, its own independent dictionary of frontier-AI-safety terms. NIKOLAI is unendorsed: it is not a standard, not an evaluator and not a regulator, and its crosswalk rows are shadow mappings — CASRAI’s reading of published framework language — unless the organisation concerned has filed a Mapping Declaration. Two N5 elements are where a control evaluation actually attaches.
Elicitation method (N5 · Evidence and evaluations) records the techniques and conditions used to draw out a model’s maximum capability during an evaluation run, together with the declared interpretation of the result — lower bound or ceiling. In a control evaluation, the attack policy is the elicitation method: the red team’s strategy, its affordances, and whether it was given protocol internals are the fields that determine whether the safety figure is a lower bound on protocol strength or an unqualified claim. Two control results are not comparable unless those fields are recorded.
Evaluation-validity threat (N5) is the named-conditions list for ways evaluation behaviour diverges from deployment behaviour — evaluation awareness, sandbagging, alignment faking, metagaming, reward hacking. Control evaluations have an awkward relationship with this element. The method exists partly because these threats make propensity evidence unreliable; but the same threats reappear inside the control evaluation, since a model that recognises it is inside a control experiment can sandbag the side task and make the protocol look stronger than it is. Our guide to evaluation-validity threats and the N5 crosswalk covers how different organisations name these conditions.
Why This Reaches Research Administration
The research-administration connection here is not a stretch, but it is CASRAI’s own extension rather than anything Redwood claims. Control evaluation is a method for a particular situation: an agent with legitimate access to your systems, more capable in its domain than your ability to check its work line by line, which you cannot simply decide to trust.
That is the exact shape of two problems research institutions already own. University research computing is now handing autonomous coding agents write access to shared clusters, pipelines and data stores — a setting where the useful question is not whether the agent is well-intentioned but whether the deployment wrapper would catch it if it were not. And research security and insider-threat programmes are, structurally, control problems: the trusted-but-weaker overseer checking a fraction of a capable insider’s work under a fixed audit budget is precisely the upfront-auditing-versus-trusted-monitoring trade the APPS results quantify.
The transferable lesson for a research computing or sponsored-programmes office is the 15%-versus-92% contrast, not the specific protocols. Spending a review budget early and then running unmonitored is the worst allocation in the table. Spreading the same budget across a suspicion-ranked stream, and having a cheap reviewer fix rather than merely flag, is the best one. That conclusion does not depend on anything about frontier models.
Frequently Asked Questions
What is an AI control evaluation?
An experimental procedure in which a red team builds an attack policy for an untrusted model and a blue team builds a deployment protocol to stop it, run across many episodes to produce a safety figure (how often the protocol prevented the harmful outcome) and a usefulness figure (how much legitimate work still got done). It measures the model’s capability to subvert the safeguards rather than its propensity to try.
How is a control evaluation different from red teaming?
Red teaming is usually an unstructured effort against a model’s safeguards, with no fixed denominator and no stopping rule. A control evaluation is a two-sided game with a defined setting, a fixed audit budget, an explicit adversary model and a stated assumption list, producing two comparable numbers. It also evaluates the whole deployment protocol, not just the model.
Who created the method?
Redwood Research, in “AI Control: Improving Safety Despite Intentional Subversion” (Greenblatt, Shlegeris, Sachan and Roger, arXiv:2312.06942, December 2023; ICML oral). Redwood has since worked with the UK AI Security Institute on control safety cases and on the ControlArena library, and states that it advises AI companies including Google DeepMind and Anthropic.
What are trusted and untrusted models?
An untrusted model is the capable system you want to use but cannot verify — it is assumed, for the evaluation, to be working against you. A trusted model is a weaker system you are willing to assume is not scheming, typically because it is too limited to execute a coherent subversion strategy. Most control protocols are arrangements for using a trusted model to oversee an untrusted one within a fixed budget.
Can a regulator rely on a control-evaluation number?
Not on the number alone. The figure is conditional on the setting, the audit budget, the red team’s affordances and an assumption list, and the translation into deployment risk is an explicit judgement step. A control result is usable as evidence when those conditions are published alongside it; quoted bare, it is a percentage without a denominator.
Does any regulation require control evaluations?
No jurisdiction currently mandates control evaluations by name. The method appears in voluntary safety-framework and safety-case literature, including work co-authored from within the UK AI Security Institute, and in shared evaluation infrastructure. Treat it as an emerging evidentiary practice rather than a compliance obligation.
Primary Sources
- Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan and Fabien Roger, “AI Control: Improving Safety Despite Intentional Subversion,” arXiv:2312.06942 (12 December 2023; v5 23 July 2024), ICML — arXiv abstract.
- Redwood Research, AI Control research page — redwoodresearch.org/research/ai-control.
- Tomek Korbak, Joshua Clymer, Benjamin Hilton, Buck Shlegeris and Geoffrey Irving, “A sketch of an AI control safety case,” arXiv:2501.17315 (28 January 2025) — arXiv abstract.
- Buck Shlegeris, “Thoughts on the conservative assumptions in AI control,” Redwood Research blog, 17 January 2025 — blog.redwoodresearch.org.
- Joshua Clymer, “Takeaways from sketching a control safety case,” Redwood Research blog, 30 January 2025 — blog.redwoodresearch.org.
- Ryan Greenblatt, “An overview of areas of control work,” Redwood Research blog, 9 April 2025 — blog.redwoodresearch.org.
- UK AI Security Institute, “Introducing ControlArena: A library for running AI control experiments,” 22 October 2025 — aisi.gov.uk; documentation at control-arena.aisi.org.uk.
- Jon Kutasov et al., “Evaluating Control Protocols for Untrusted AI Agents,” arXiv:2511.02997 (4 November 2025) — arXiv abstract.
CASRAI is not affiliated with Redwood Research, the UK AI Security Institute, or any organisation named on this page. NIKOLAI is CASRAI’s own independent dictionary and is not endorsed by the organisations whose framework language it maps.
