Skip to main content
v2026.11,986 entries · CC-BY 4.0
CASRAIAsk CASRAI · AI assistant for researchAsk a research question. Answers come with their sources.Funder rules, compliance, methods and publishing. Answers link the sources they rest on.
NIKOLAI elementN2 · Threat models and risk framingProposednikolai-v0.3

Threat Model

Definition

A threat model is a structured statement of how an AI system could contribute to a severe harm: who might cause it, by what means, what role the AI plays and how large the harm could be. Safety frameworks use threat models to decide which capabilities to measure and which safeguards to require.

Also called: risk model, systemic risk model

NIKOLAI's proposed operational definition

NIKOLAI editorial proposal (unsourced): a threat-model record is the structured statement of who could cause which harm, by what means, with what role played by the AI system, at what magnitude of harm, together with the developer's stated rationale for prioritising that threat model over others considered.

The operational definition is CASRAI's own proposal, not a definition any named organisation has agreed to. See what NIKOLAI is and is not, or browse the AI safety glossary.

Examples

Examples from published documents

  • Anthropic's CB-1 threat model: "Individuals or small groups with limited resources use AI models to gain access to non-novel chemical or biological (CB) weapons, leading to the risk of catastrophic harm" (Anthropic Risk Report, August 2026, section 4.1)
  • Microsoft's risk models "are designed to prioritize specific scenarios, threat actors, unique ways an AI model may increase or decrease risk, technical bottlenecks, and the severity and likelihood of the scenario resulting in significant impacts" (Microsoft Frontier Governance Framework, section 2)
  • OpenAI separates two ways a risk can be realised: "a malicious user, who can leverage the model to cause the severe harm, and a misaligned model, which autonomously causes the harm" (OpenAI Preparedness Framework v2, section 4.1)

Counter-examples

Not to be confused with

  • Threat modelling is the process; the threat model is its output. Meta defines threat modelling as "a structured process of identifying how Frontier AI could contribute to specific" catastrophic outcomes.
  • A security threat model for protecting model weights is a different object: Anthropic's RSP changelog treats "ASL-3 and ASL-2 security threat models" (which attackers are in scope) separately from its catastrophic-harm threat models.

Related terms

Related AI safety terms

Terms defined under this element that are not NIKOLAI elements themselves, each as the cited document uses it. Every term is also listed in the AI safety glossary.

Misuse
Misuse is harm caused when people deliberately use an AI model's capabilities to do damage. Google DeepMind's FSF v3.1 defines misuse risk as "risks of threat actors using critical capabilities of deployed or exfiltrated models to cause harm".
Source: Google DeepMind Frontier Safety Framework v3.1
Misalignment
Misalignment, in Anthropic's August 2026 Risk Report, is a model computation that a fully informed reasonable person would judge unethical, illegal, clearly objectionable or inconsistent with the model's constitution, and that influences its output.
Source: Anthropic Risk Report, August 2026
Loss of control
Loss of control is the risk that humans lose the ability to reliably direct, modify or shut down an AI model. The EU Code of Practice uses this wording; Meta's framework adds that humans "cannot feasibly regain" that ability.
Source: EU GPAI Code of Practice, Safety and Security Chapter
Automated AI R&D
Automated AI R&D is AI doing the work of AI researchers. Anthropic's RSP v3.4 covers systems that "fully automate, or otherwise dramatically accelerate, the work of large, top-tier teams of human researchers"; Google DeepMind's levels are called ML R&D acceleration and automation.
Source: Anthropic Responsible Scaling Policy v3.4
Recursive self-improvement
Recursive self-improvement is AI systems speeding up the development of more capable AI systems, which then speed it up further. Anthropic says its automated R&D threshold "is intended to capture the onset of dramatic recursive self-improvement"; OpenAI tracks "AI Self-improvement".
Source: Anthropic Responsible Scaling Policy v3.4

Source of record

Where this definition comes from

Crosswalk

How named organisations use this concept

Every row below is a shadow mapping. A shadow row is CASRAI's own reading of a published document. No lab, evaluator or regulator named on a shadow row has declared, endorsed, or been consulted on it. That changes only when an organisation files its own Mapping Declaration.
OrganisationTheir term, as publishedMatch & verificationSource
AnthropicShadow mapping
Anthropic Risk Report, August 2026
“"CB-1 threat model": "Individuals or small groups with limited resources use AI models to gain access to non-novel chemical or biological (CB) weapons, leading to the risk of catastrophic harm" (§4.1, §4.2.1). Prioritisation criteria: expected damages; "a clear role for AI in creating risk beyond what is created by other technologies and background conditions"; historical sanity checks; generalisability and poor early warning (§6.1).”exactEQ
confidence: high
Anthropic Risk Report, August 2026
OpenAIShadow mapping
OpenAI Preparedness Framework v2
“"Threat model": for each Tracked Category, "identifying specific risks of severe harms that could arise from the frontier capabilities in that domain and sets corresponding capability thresholds" (§2.2, p.4); "SAG reviews and approves these threat models."”closeCL
confidence: high
OpenAI Preparedness Framework v2
Google DeepMindShadow mapping
Gemini 3.7 Flash FSF report
“Report method terms: "Threat actor type", "Scenario" ("combinations of threat actor types and agents, weapons, or attacks"), "Harm journey" ("A breakdown of key stages and substages required to carry out such an attack end-to-end"), "Bottleneck sub-stages", "Web-only baseline" (report, p.8).”closeCL
confidence: high
Gemini 3.7 Flash FSF report
xAIShadow mapping
xAI Frontier AI Framework (30 Jun 2026)
“"xAI has developed systemic risk scenarios that enumerate the causal factors, potential harms, and mitigations" (s.2.1), which are not published.”
Source PDF's own metadata /Title reads "Privileged/Confidential DRAFT working FRAMEWORK DOC"; no xAI statement disambiguating draft vs. final was found. Any claim built on this row should carry that caveat forward.
noneDU
confidence: medium
xAI Frontier AI Framework (30 Jun 2026, draft-marked)
MetaShadow mapping
Meta Advanced AI Scaling Framework v2
“"Threat modeling: a structured process of identifying how Frontier AI could contribute to specific ... outcomes"; "Threat scenarios: describe the real-world events ... including enabling capabilities, deployment context, and threat actors (as relevant) ... that may be sufficient to produce a catastrophic outcome"; identifiers such as "Cyber 1" and "TS.1.1" (Appendix I; §3.4).”exactEQ
confidence: high
Meta Advanced AI Scaling Framework v2
EUShadow mapping
EU GPAI Code of Practice, Safety and Security chapter
“Measure 2.2 "Systemic risk scenarios": "Signatories will develop appropriate systemic risk scenarios ... for each identified systemic risk," feeding Measure 3.3 "systemic risk modelling." No named identifier scheme and no publication requirement beyond the Model Report; confirms xAI's own footnote claim that its unpublished scenarios follow this chapter.”
Row also cites xAI's Frontier AI Framework, whose PDF metadata is marked "DRAFT working FRAMEWORK DOC" with no disambiguating xAI statement found.
closeCL
confidence: high
EU GPAI Code of Practice, Safety and Security chapter
Frontier Model ForumShadow mapping
FMF Risk Taxonomy and Thresholds
“"Threat modeling: a process for systematically anticipating and identifying how various threat actors might leverage frontier AI to achieve harmful outcomes and mapping the potential pathways to those outcomes." "Threat scenarios": scenarios "that specify how adversaries might use frontier AI to achieve severe outcomes, identifying specific tasks, model capabilities to exploit, and complementary tools." (s2.1, pp.7-8)”exactEQ
confidence: high
FMF Risk Taxonomy and Thresholds
MicrosoftShadow mapping
Microsoft Frontier Governance Framework (Feb 2026)
“"For each tracked high-risk capability, we conduct risk modeling to guide our red teaming and evaluation practices. These risk models, informed by external experts, are designed to prioritize specific scenarios, threat actors, unique ways an AI model may increase or decrease risk, technical bottlenecks, and the severity and likelihood of the scenario resulting in significant impacts." (section 2, p.4)”closeCL
confidence: high
Microsoft Frontier Governance Framework (Feb 2026)
What do these codes mean?
exact
The source term is equivalent to this element
close
The source term is close but not equivalent to this element
broad
The source term is broader than this element
narrow
The source term is narrower than this element
none
No mapping claim — used for false-friend and declared-but-undefined rows
EQ
Equivalent
CL
Close
BR
Source is broader than the element
NR
Source is narrower than the element
FF
False friend — same or similar label, different meaning
DU
Declared but undefined by the source
UV
Unverified

Gap

Among the sources reviewed, public identifiers for individual threat scenarios (such as "Cyber 1" and "TS.1.1") were found in Meta's framework, and Meta withholds "full details of the constituent steps and tasks within a threat scenario" (§3.4). NIKOLAI should separate a public identifier and summary from restricted detail; see the element threat-scenario.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →