Examples
Examples from published documents
- Microsoft admits a benchmark to its leading-indicator suite only if it has low saturation, measures an advanced capability such as mathematical reasoning, and has enough prompts to account for non-determinism in model output (Microsoft Frontier Governance Framework, footnote 1, p.5)
- G42 first screens a model on open-source and in-house benchmarks, and presumes it is below a capability threshold if it scores lower than an outside model already judged to be below that threshold (G42 Frontier Safety Framework, PDF p.4)
- OpenAI's MLE-Bench Revised keeps the highest-signal tasks from the original MLE-Bench and replaces saturated or low-signal tasks with new ML engineering problems from 2025 and 2026 (GPT-5.6 system card, section 9.1.3)
Counter-examples
Not to be confused with
- An uplift study is not a benchmark: it measures people's performance with and without AI access in a controlled trial, not a model's score on a fixed task set.
- A red-team exercise adapts its attacks to the model as testing goes on; a benchmark stays fixed so that results are comparable across models.
Related terms
Related AI safety terms
Terms defined under this element that are not NIKOLAI elements themselves, each as the cited document uses it. Every term is also listed in the AI safety glossary.
- Contamination
- Contamination is when evaluation items, or close copies of them, end up in an AI model's training data, so the model may score well by recall rather than ability; OpenAI describes contaminated evals as ones the model saw during training.
- Source: GPT-5.6 deployment safety card
- Held-out test set
- A held-out test set is evaluation material deliberately kept away from model training and from the teams doing the training, so that scores reflect general ability rather than overfitting to the test.
- Source: Meta Advanced AI Scaling Framework v2
- Canary string
- A canary string is a unique marker placed in published evaluation data so that AI developers can detect it and exclude that data from training, keeping test scores from being inflated by memorisation.
- Source: Anthropic Risk Report (August 2026)
- Evaluation suite
- An evaluation suite is a named collection of related tasks or benchmarks run together and reported as a group, revised over time as tasks saturate, prove flawed or are replaced with harder ones.
- Source: GPT-5.6 deployment safety card
Source of record
Where this definition comes from
Anthropic Institute, "Recursive self-improvement", main text
“Benchmarks measure the performance of models in a given domain, and they're 'saturated' when models achieve close to 100% performance.”
https://www.anthropic.com/institute/recursive-self-improvementMicrosoft Frontier Governance Framework, fn 1, p.5
“For a benchmark to be included in our suite of leading indicator assessments it must: 1) have low saturation (i.e., the best performing models typically score lower than 70%); 2) measure an advanced capability, for example, mathematical reasoning, rather than an application-oriented capability like financial market prediction; and 3) have a sufficient number of prompts to account for non-determinism in model output.”
https://cdn-dynmedia-1.microsoft.com/is/content/microsoftcorp/microsoft/msc/documents/presentations/CSR/Frontier-Governance-Framework-Feb-2026.pdfG42 Frontier Safety Framework, Frontier Capability Threshold evaluations, PDF p.4
“At the outset, G42 will conduct preliminary evaluations based on open-source and in-house benchmarks relevant to a hazardous capability.”
https://www.g42.ai/application/files/9517/3882/2182/G42_Frontier_Safety_Framework_Publication_Version.pdf
Crosswalk
How named organisations use this concept
| Organisation | Their term, as published | Match & verification | Source |
|---|---|---|---|
| AnthropicShadow mapping Anthropic Institute, "Recursive self-improvement" | “"Benchmarks measure the performance of models in a given domain, and they're 'saturated' when models achieve close to 100% performance."” | exactEQ confidence: high | Anthropic Institute, "Recursive self-improvement" |
| MicrosoftShadow mapping Microsoft Frontier Governance Framework | “Benchmarks are used as leading indicators; inclusion requires "low saturation (i.e., the best performing models typically score lower than 70%)", measurement of "an advanced capability", and "a sufficient number of prompts to account for non-determinism in model output" (fn 1, p.5).” | closeCL confidence: high | Microsoft Frontier Governance Framework |
| G42Shadow mapping G42 Frontier Safety Framework | “"preliminary evaluations based on open-source and in-house benchmarks relevant to a hazardous capability"; a model scoring below an outside model already judged below the threshold "will be presumed to be below the capability threshold" (PDF p.4).” | closeCL confidence: high | G42 Frontier Safety Framework |
| MetaShadow mapping Meta Advanced AI Scaling Framework v2 | “Preparedness reports "will provide sufficient methodological detail on any open-source benchmarks included in the report" (§2.2.1, p.8); acceleration signal: "the model saturates benchmarks in each domain six months or less after public release of a benchmark" (fn 8, p.35).” | noneDU confidence: high | Meta Advanced AI Scaling Framework v2 |
| Google DeepMindShadow mapping Frontier Safety Framework v3.1 | “Capability change assessments draw on "performance on general capability benchmarks"; light-weight early warning evaluations include "our automated benchmarks" (§1.3.2, p.5).” | noneDU confidence: high | Google DeepMind Frontier Safety Framework v3.1 |
| EUShadow mapping EU GPAI Code of Practice, Safety and Security Chapter | “"Examples of model evaluation methods are: Q&A sets, task-based evaluations, benchmarks, red-teaming and other methods of adversarial testing, human uplift studies, model organisms, simulations, and/or proxy evaluations for classified materials." (Measure 3.2)” | noneDU confidence: high | EU GPAI Code of Practice, Safety and Security Chapter |
| OpenAIShadow mapping OpenAI Preparedness Framework v2 | “"Scalable Evaluations: automated evaluations designed to measure proxies that approximate whether a capability threshold has been crossed", with "indicative thresholds" set in advance (§3.1).” OpenAI's scalable evaluations are automated, repeatable tests like benchmarks, but the framework does not call them benchmarks. | closeCL confidence: medium | OpenAI Preparedness Framework v2 |
What do these codes mean?
- exact
- The source term is equivalent to this element
- close
- The source term is close but not equivalent to this element
- broad
- The source term is broader than this element
- narrow
- The source term is narrower than this element
- none
- No mapping claim — used for false-friend and declared-but-undefined rows
- EQ
- Equivalent
- CL
- Close
- BR
- Source is broader than the element
- NR
- Source is narrower than the element
- FF
- False friend — same or similar label, different meaning
- DU
- Declared but undefined by the source
- UV
- Unverified
Related, not mapped
Pointers that are not crosswalk claims
These sources mention this concept but do not define or map it clearly enough to count as a crosswalk row — noted here so the research is visible without overstating it as a mapping.
- Demis Hassabis (personal essay)
Proposes a standards body that would "eventually" build "its own held-out tests independent of the" labs, with evaluations "regularly updated, perhaps quarterly to start" - a pointer to benchmark governance, not a benchmark definition.
Demis Hassabis, substack post - METR
Time Horizon 1.1 is described as following "the same methodology described in the initial paper, but with a larger task suite" - a versioned task suite, recorded here as a pointer.
METR
Divergence
Where sources materially disagree
Sources give benchmarks different jobs. Microsoft uses general-purpose benchmarks as leading indicators and admits only those where top models score under 70%; G42 uses open-source and in-house benchmarks as a preliminary screen before in-depth evaluations; OpenAI pairs automated evaluations with pre-set indicative thresholds; Anthropic reports that its task-based AI R&D evaluation suite has reached the point where frontier models surpass human baseline performance on most tasks, and has added an evaluation built on real internal tasks.
Gap
The research did not find a shared registry of safety-relevant benchmarks recording version, provenance and contamination status; each developer names its own suite and versions it independently.
