Risk Prioritisation Criteria in AI Safety

Risk Prioritisation Criteria

Definition

Risk prioritisation criteria are the tests a developer uses to decide which possible AI harms its safety framework will track. Common tests ask whether a harm is plausible, severe, newly made possible by AI and hard to reverse or to see coming.

Also called: threat model criteria, inclusion criteria, criteria for inclusion and omission

NIKOLAI's proposed operational definition

NIKOLAI editorial proposal (unsourced): risk prioritisation criteria are recorded as a rule: the named tests, in the source's own words, that a framework applies to decide whether a candidate risk enters its scope. They are recorded criterion by criterion so a reader can see which tests each covered or excluded risk passed or failed.

The operational definition is CASRAI's own proposal, not a definition any named organisation has agreed to. See what NIKOLAI is and is not, or browse the AI safety glossary.

Examples

Examples from published documents

OpenAI's v2 framework states: "Going forward we will handle risks related to persuasion outside the Preparedness Framework", after narrowing its Tracked Categories to biological and chemical, cybersecurity and AI self-improvement (OpenAI Preparedness Framework v2, Changes in version 2)
Meta keeps radiological and nuclear risks as an emerging area because "materials access is a strong bottleneck to nuclear risk" (Meta Advanced AI Scaling Framework v2, section 4.2.4)

Counter-examples

Not to be confused with

Capability thresholds are not prioritisation criteria: the criteria decide which risks a framework covers at all, while thresholds decide when a covered risk calls for stronger safeguards.

Related terms

Related AI safety terms

Terms defined under this element that are not NIKOLAI elements themselves, each as the cited document uses it. Every term is also listed in the AI safety glossary.

Heightened marginal risk: Heightened marginal risk is the FMF's first test for including a risk in a framework: the model "creates heightened risk, not merely a similarly effective alternative to existing tools (e.g., search engines)", for example by democratising dangerous capabilities.; Source: FMF Risk Taxonomy and Thresholds
Expected damages: Expected damages, as Anthropic's August 2026 Risk Report uses the term, are "the probability of each potential harm times the size of the harm, summed over all potential harms". Anthropic ranks threat models partly on rough estimates of them.; Source: Anthropic Risk Report, August 2026

Source of record

Where this definition comes from

Anthropic Risk Report, August 2026, §6.1
“"Our criteria for prioritizing threat models are:" "A combination of high potential damages and high likelihood for these potential damages." "A clear role for AI in creating risk beyond what is created by other technologies and background conditions." "Threat scenarios that survive sanity checks considering historical analogies"”
https://www-cdn.anthropic.com/f61d49fa5596956a5dec75fea0e973bf6a6a8378/Redacted%20Risk%20Report%20August%202026%20.pdf
OpenAI Preparedness Framework v2, §2.1
“"We treat a frontier capability as a Tracked Category if the capability creates a risk that meets five criteria": "Plausible", "Measurable", "Severe", "Net new", "Instantaneous or irremediable".”
https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf

Crosswalk

How named organisations use this concept

Every row below is a shadow mapping. A shadow row is CASRAI's own reading of a published document. No lab, evaluator or regulator named on a shadow row has declared, endorsed, or been consulted on it. That changes only when an organisation files its own Mapping Declaration.

Organisation	Their term, as published	Match & verification	Source
AnthropicShadow mapping Anthropic Risk Report, August 2026	“"Our criteria for prioritizing threat models are": "A combination of high potential damages and high likelihood for these potential damages"; "A clear role for AI in creating risk beyond what is created by other technologies and background conditions"; "Threat scenarios that survive sanity checks considering historical analogies"; and additional factors including "the extent to which threats are hard to get early warning signs of and respond to accordingly" (§6.1).”	exactEQ confidence: high	Anthropic Risk Report, August 2026
OpenAIShadow mapping OpenAI Preparedness Framework v2	“Five criteria for a Tracked Category: "1. Plausible: It must be possible to identify a causal pathway for a severe harm in the capability area, enabled by frontier AI. 2. Measurable ... 3. Severe ... 4. Net new ... 5. Instantaneous or irremediable" (§2.1). Footnote 3: "These criteria were informed in part by Meta's recent Frontier AI Framework."”	exactEQ confidence: high	OpenAI Preparedness Framework v2
MetaShadow mapping Meta Advanced AI Scaling Framework v2	“"we include a set of criteria for inclusion and omission below"; "we seek to consider risks that satisfy all four criteria": Plausible, Catastrophic, Net new, Instantaneous or irremediable (§3.2, pp.13-14).”	exactEQ confidence: high	Meta Advanced AI Scaling Framework v2
Frontier Model ForumShadow mapping FMF Risk Taxonomy and Thresholds	“"Frontier AI frameworks are especially valuable for risks that share the following features: 1. Heightened Marginal Risk ... 2. Severity and Scale ... 3. Credibility ... 4. Velocity and Irreversibility" (s1.3, pp.4-5).” A cross-member synthesis, not one developer's own criteria.	exactEQ confidence: high	FMF Risk Taxonomy and Thresholds
Google DeepMindShadow mapping Google DeepMind Frontier Safety Framework v3.1	“"We identify potential risks that could stem from our models and analyze their characteristics to determine which of the identified risks could be significant or severe risks." (s.1.3.1)” Selection is described, but no named list of criteria is published.	broadBR confidence: medium	Google DeepMind Frontier Safety Framework v3.1
EUShadow mapping EU GPAI Code of Practice, Safety and Security Chapter	“Measure 2.1: "(b) analysing relevant characteristics of the risks compiled pursuant to point (a), such as their nature (based on Appendix 1.2) and sources (based on Appendix 1.3); and (c) identifying, based on point (b), the systemic risks stemming from the model".” The characteristics in Appendix 1.2 (e.g. capability-dependent, high velocity, difficult or impossible to reverse) serve as the selection tests.	closeCL confidence: high	EU GPAI Code of Practice, Safety and Security Chapter

What do these codes mean?

exact: The source term is equivalent to this element
close: The source term is close but not equivalent to this element
broad: The source term is broader than this element
narrow: The source term is narrower than this element
none: No mapping claim — used for false-friend and declared-but-undefined rows
EQ: Equivalent
CL: Close
BR: Source is broader than the element
NR: Source is narrower than the element
FF: False friend — same or similar label, different meaning
DU: Declared but undefined by the source
UV: Unverified

Divergence

Where sources materially disagree

OpenAI's five criteria were "informed in part by Meta's recent Frontier AI Framework", and the two lists share Plausible, Net new and Instantaneous or irremediable. OpenAI adds Measurable, and Meta's Catastrophic stands where OpenAI has Severe. Anthropic's criteria rank threat models rather than exclude them, and add historical sanity checks and early-warning difficulty. The FMF's four features are a synthesis across members, and its Heightened Marginal Risk feature ties selection to a baseline (see the N4 element marginal-absolute-risk-basis).