Examples
Examples from published documents
- Anthropic counts its AI R&D threshold as met if models could fully substitute for all its Research Scientists and Research Engineers at competitive cost, within a factor of 5, or if AI progress shows dramatic acceleration (Anthropic RSP v3.4, section 1)
- Meta concludes a model does not exceed the moderate-or-lower cyber risk level if it achieves "< 75% pass@10 success on all categories of simple capture the flag (CTF) challenges in unguided mode" (Meta Advanced AI Scaling Framework v2, §4.2.1)
- For ML R&D automation, Google DeepMind notes a model would need to complete 100% of benchmark challenges to reach the level but "we conservatively set our safety margin at 90%" (Gemini 3.7 Flash FSF report, p.4)
Counter-examples
Not to be confused with
- The descriptive wording of a threshold, such as "AI systems that can functionally substitute for the scarce human expertise" in weapons development, is the threshold itself, not its operationalisation.
Related terms
Related AI safety terms
Terms defined under this element that are not NIKOLAI elements themselves, each as the cited document uses it. Every term is also listed in the AI safety glossary.
- Indicative threshold
- An indicative threshold, in OpenAI's Preparedness Framework, is a pre-set score on an automated evaluation that indicates a model may have reached a capability threshold. It informs, but does not by itself decide, the threshold determination.
- Source: OpenAI Preparedness Framework v2
Source of record
Where this definition comes from
Anthropic Responsible Scaling Policy v3.4, Section 1 table, p.9
“"We will consider this threshold to be met if we determine that either (1) our models would be able to fully substitute for our entire set of Research Scientists and Research Engineers, at competitive costs (i.e., within a factor of 5); or (2) there is “dramatic acceleration” of the pace of AI progress for reasons that likely relate to the automation of AI R&D."”
https://cdn.sanity.io/files/4zrzovbb/website/0bacdc8440ea96e62a8766d99ebe1d4eea6d5f3a.pdfOpenAI Preparedness Framework v2, §3.1
“"Scalable Evaluations: automated evaluations designed to measure proxies that approximate whether a capability threshold has been crossed. Scalable evaluations have associated “indicative thresholds,” which are levels of performance that we have pre-determined to indicate that a deployment may have reached a capability threshold."”
https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf
Crosswalk
How named organisations use this concept
| Organisation | Their term, as published | Match & verification | Source |
|---|---|---|---|
| AnthropicShadow mapping Responsible Scaling Policy v3.4 | “AI R&D test: "We will consider this threshold to be met if we determine that either (1) our models would be able to fully substitute for our entire set of Research Scientists and Research Engineers, at competitive costs (i.e., within a factor of 5); or (2) there is “dramatic acceleration” of the pace of AI progress for reasons that likely relate to the automation of AI R&D." (p.9). Changelog v3.1: "how we operationalize the Automated R&D capability threshold"; v3.4: the threshold "has proven difficult to operationalize."” | exactEQ confidence: high | Anthropic Responsible Scaling Policy v3.4 |
| OpenAIShadow mapping Preparedness Framework v2 | “"Scalable Evaluations: automated evaluations designed to measure proxies that approximate whether a capability threshold has been crossed. Scalable evaluations have associated “indicative thresholds,” which are levels of performance that we have pre-determined to indicate that a deployment may have reached a capability threshold." (§3.1). "The determination that a threshold has been reached is informed by these indicative results from capability evaluations, and also reflects holistic judgment based on the totality of available evidence" (§3.3).” The indicative threshold values themselves are not published in the framework. | exactEQ confidence: high | OpenAI Preparedness Framework v2 |
| MetaShadow mapping Meta Advanced AI Scaling Framework v2 | “"if a model under test achieves < 75% pass@10 success on all categories of simple capture the flag (CTF) challenges in unguided mode, we conclude that it does not exceed the 'moderate or lower' risk level." (§4.2.1, p.28). "As an illustrative example, our risk acceptance criteria include thresholds such as at least 40% on MASK and at most 50% on Agent Misalignment." (§4.2.3)” Meta's cut-offs operationalise risk levels by deployment scenario rather than capability thresholds in the narrow sense; the MASK criteria are labelled illustrative. | closeCL confidence: high | Meta Advanced AI Scaling Framework v2 |
| MagicShadow mapping Magic AGI Readiness Policy | “"when, at the end of a training run, our models exceed a threshold of 50% accuracy on LiveCodeBench, we will trigger our commitment to incorporate a full system of dangerous capabilities evaluations and planned mitigations into our AGI Readiness Policy" (Threshold Definition)” The page carries the banner "Note: This policy is outdated. We're working on an update." | exactEQ confidence: high | Magic AGI Readiness Policy |
| Google DeepMindShadow mapping Gemini 3.7 Flash FSF report / Frontier Safety Framework v3.1 | “ML R&D automation: "a model theoretically needs to complete 100% of the benchmark challenges to reach the CCL, but we conservatively set our safety margin at 90%" (report p.4). FSF v3.1: alert thresholds "may draw on evaluation results, expert assessments, and other sources of information" (s.1.3.2).” Numeric values are disclosed for this one capability level only; no numeric alert-threshold values are given for any domain. | closeCL confidence: high | Gemini 3.7 Flash FSF report |
| G42Shadow mapping G42 Frontier Safety Framework | “Presumption rule for preliminary evaluations: "If a given G42 model achieves lower performance on relevant open-source benchmarks than a model produced by an outside organization that has been evaluated to be definitively below the capability threshold, then such G42 model will be presumed to be below the capability threshold." (s.2)” | closeCL confidence: medium | G42 Frontier Safety Framework |
| EUShadow mapping EU GPAI Code of Practice, Safety and Security chapter | “Measure 4.1: Signatories "define appropriate systemic risk tiers that: (i) are defined in terms of model capabilities, and may additionally incorporate model propensities, risk estimates, and/or other suitable metrics; (ii) are measurable; and (iii) comprise at least one systemic risk tier that has not been reached by the model".” Requires tiers to be measurable but leaves the measurement to each Signatory. | broadBR confidence: high | EU GPAI Code of Practice, Safety and Security chapter |
| AI Seoul Summit (Frontier AI Safety Commitments)Shadow mapping Frontier AI Safety Commitments | “Footnote 3: "It should be possible to assess whether thresholds have been breached."” A requirement that thresholds be assessable, with no method specified. | broadBR confidence: high | Frontier AI Safety Commitments (AI Seoul Summit 2024) |
What do these codes mean?
- exact
- The source term is equivalent to this element
- close
- The source term is close but not equivalent to this element
- broad
- The source term is broader than this element
- narrow
- The source term is narrower than this element
- none
- No mapping claim — used for false-friend and declared-but-undefined rows
- EQ
- Equivalent
- CL
- Close
- BR
- Source is broader than the element
- NR
- Source is narrower than the element
- FF
- False friend — same or similar label, different meaning
- DU
- Declared but undefined by the source
- UV
- Unverified
Related, not mapped
Pointers that are not crosswalk claims
These sources mention this concept but do not define or map it clearly enough to count as a crosswalk row — noted here so the research is visible without overstating it as a mapping.
- Amazon
Amazon's Critical Capability Thresholds are stated qualitatively, without benchmarks, scores or pass/fail criteria; terms such as "material uplift" and "reliably" are not operationalised in the published framework.
Amazon Frontier Model Safety Framework
Divergence
Where sources materially disagree
Two approaches coexist. Benchmark-style operationalisation fixes a metric and cut-off in advance (Magic's 50% on LiveCodeBench, Meta's < 75% pass@10, OpenAI's indicative thresholds). Argument-style operationalisation deliberately avoids fixed tests: Anthropic's RSP v2.0 changelog says "Rather than using prespecified evaluations, we now require an affirmative case that models are sufficiently far from Capability Thresholds", and RSP v3.4 says it cannot "give highly specific advance detail on what evaluations will determine whether risk thresholds have been passed." Even benchmark-style cut-offs are often indicators feeding a holistic judgment (OpenAI §3.3) rather than decision rules.
Gap
Most frameworks publish the descriptive threshold but not the operational test: OpenAI's indicative threshold values are not published, Google DeepMind gives no numeric alert-threshold values, and Amazon and xAI publish no cut-offs. The research found no shared format for recording an operationalisation with its version and conditions.
