Skip to main content
v2026.11,986 entries · CC-BY 4.0
CASRAIAsk CASRAI · AI assistant for researchAsk a research question. Answers come with their sources.Funder rules, compliance, methods and publishing. Answers link the sources they rest on.
NIKOLAI elementN7 · IncidentsProposednikolai-v0.3

Incident

Definition

An incident, in frontier-AI safety reporting, is an event in which the development, use or malfunction of an AI model causes harm, or a safety or security failure that could lead to harm, such as unauthorised access to model weights or a model evading its developer's controls.

Also called: Critical safety incident, Serious incident, AI incident, AI safety incident, Critical incident, Major incident

NIKOLAI's proposed operational definition

NIKOLAI editorial proposal (unsourced, not attributed to any lab or regulator): an Incident is a record of an event, identified by a persistent identifier, in which a model, safeguard, or organizational process produced or nearly produced a harm or a safety-relevant failure. The record carries at minimum: start/end dates, a scope description, a current status, the model or system involved, a description of its involvement, and any corrective or response measures taken. NIKOLAI does not itself define what threshold of severity makes an event reportable -- that varies by source (see crosswalk) -- only the record shape needed to compare across sources.

The operational definition is CASRAI's own proposal, not a definition any named organisation has agreed to. See what NIKOLAI is and is not, or browse the AI safety glossary.

Examples

Examples from published documents

  • Under California law a critical safety incident includes "Loss of control of a frontier model causing death or bodily injury" and a frontier model using "deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior" (California SB 53, Cal. Bus. & Prof. Code 22757.11(d))
  • In July 2026 a review of 141,006 evaluation runs found "three incidents in which a model accessed the internet from within or while interacting with the evaluation environment" and then gained unauthorised access to real organisations' systems (Anthropic, Investigating three real-world incidents in our cybersecurity evaluations)
  • On 28 July 2026 AISI's security monitoring flagged data leaving a testing system; AISI "declared a security incident, stopped all related evaluations" and found agents had taken unsanctioned actions against real people and organisations (UK AI Security Institute, Incident Report, 4 Aug 2026)

Counter-examples

Not to be confused with

  • A hazard or near miss is not an incident: the OECD defines an AI hazard as an event that "could plausibly lead to an AI incident", and the EU Code of Practice treats a near miss as a serious incident that "could have, but ultimately did not, materialise".
  • A capability-threshold crossing found in a pre-deployment evaluation is not by itself an incident; SB 53's deception limb expressly excludes behaviour inside "an evaluation designed to elicit this behavior".

Related terms

Related AI safety terms

Terms defined under this element that are not NIKOLAI elements themselves, each as the cited document uses it. Every term is also listed in the AI safety glossary.

Critical safety incident
Critical safety incident is the reportable-event term in California SB 53 and New York's RAISE Act: unauthorised weight access causing death or injury, harm from a materialised catastrophic risk, loss of control causing death or injury, or deceptive evasion of developer controls raising catastrophic risk.
Source: California SB 53 (Transparency in Frontier Artificial Intelligence Act), Chapter 138, Statutes of 2025
Serious incident
Serious incident is the EU AI Act's term (Article 3(49)) for an AI incident or malfunction leading to death or serious harm to health, serious and irreversible disruption of critical infrastructure, infringement of fundamental-rights law, or serious harm to property or the environment.
Source: EU AI Act (Regulation (EU) 2024/1689), Article 3
AI incident
AI incident, in the OECD's definition, is an event, circumstance or series of events where the development, use or malfunction of one or more AI systems directly or indirectly leads to harm to health, critical infrastructure, human rights, property, communities or the environment.
Source: OECD, Defining AI incidents and related terms (OECD Artificial Intelligence Papers No. 16, 2024)
AI safety incident
AI safety incident, in US Senate bill S. 5061 (introduced 2026, not enacted), is an event that materially increases the risk that operating an AI system leads to a state in which human life, health, property or the environment is endangered.
Source: S. 5061 (119th Congress)
Serious AI incident
Serious AI incident is the OECD's term for the subset of AI incidents that lead to death or serious harm to health, serious and irreversible disruption of critical infrastructure, serious human-rights violations, or serious harm to property, communities or the environment.
Source: OECD, Defining AI incidents and related terms (OECD Artificial Intelligence Papers No. 16, 2024)
AI disaster
AI disaster, in the OECD's terminology, is a serious AI incident that disrupts the functioning of a community or a society and may test or exceed its capacity to cope using its own resources.
Source: OECD, Defining AI incidents and related terms (OECD Artificial Intelligence Papers No. 16, 2024)
Incident response
Incident response is the coordinated process of containing, mitigating and recovering from an incident once it has been identified. It is real-time and operational, and is led mainly by the affected developer's technical, legal and security teams.
Source: Frontier Model Forum -- Information Sharing Issue Brief
Incident response plan
Incident response plan is a developer's documented procedure for identifying and responding to incidents. OpenAI's AI Safety Incident Response Plan, for example, sets steps for detection, triage, severity, investigation, mitigation and deciding whether an incident must be reported externally.
Source: OpenAI Frontier Governance Framework

Source of record

Where this definition comes from

  • EU GPAI Code of Practice, Safety and Security Chapter, Commitment 9 / Measure 9.2

    “Signatories track, document and report "relevant information about serious incidents along the entire model lifecycle and possible corrective measures". Measure 9.2 lists nine required fields (start/end dates; "the resulting harm and the victim or affected group"; chain of events; "the model involved"; description of the model's involvement; response taken; recommendation to authorities; "a root cause analysis ..., including ... any failures or circumventions of systemic risk mitigations"; related post-market-monitoring patterns).”

    https://ec.europa.eu/newsroom/dae/redirection/document/118119
  • California SB 53 (Transparency in Frontier Artificial Intelligence Act), Chapter 138, Statutes of 2025, Cal. Bus. & Prof. Code 22757.11(d)

    “"Critical safety incident" means any of the following: (1) Unauthorized access to, modification of, or exfiltration of, the model weights of a frontier model that results in death or bodily injury. (2) Harm resulting from the materialization of a catastrophic risk. (3) Loss of control of a frontier model causing death or bodily injury. (4) A frontier model that uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk.”

    https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260SB53
  • Anthropic Advanced AI Framework, p.4 (definitions)

    “"Critical Safety Incident. Refers to any of the following: (a) unauthorized exfiltration of a covered model's weights, or unauthorized, deliberate, malicious modification of a covered model's weights; (b) harm resulting from the materialization of a Catastrophic Risk; (c) loss of control of a model causing death or bodily injury; (d) a model that uses deceptive techniques against the Covered Developer to subvert its controls or monitoring outside the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased Catastrophic Risk."”

    https://www-cdn.anthropic.com/files/4zrzovbb/website/0a58d567024a8b448ff15158ebc3625328dfcc1f.pdf

Crosswalk

How named organisations use this concept

Every row below is a shadow mapping. A shadow row is CASRAI's own reading of a published document. No lab, evaluator or regulator named on a shadow row has declared, endorsed, or been consulted on it. That changes only when an organisation files its own Mapping Declaration.
OrganisationTheir term, as publishedMatch & verificationSource
AnthropicShadow mapping
Anthropic Advanced AI Framework; Anthropic Risk Report (August 2026)
“AAF defines "Critical Safety Incident" in four limbs (p.4). The Risk Report keeps an incident register in §4.5.8, which "tries to paint a reasonably comprehensive picture of notable risk acceptances or safety-related incidents pertaining to our blocking bio classifiers and access controls", and a separate list of "Safety process failures": "historical cases where Anthropic's safety and security posture fell short of our ideal in some way" (§5.2.1).”
The definition is from the Advanced AI Framework (p.4), a legislative proposal; the two register quotes are from Anthropic's Risk Report (August 2026), §5.2.1: https://www-cdn.anthropic.com/f61d49fa5596956a5dec75fea0e973bf6a6a8378/Redacted%20Risk%20Report%20August%202026%20.pdf
closeCL
confidence: medium
Anthropic Advanced AI Framework
OpenAIShadow mapping
OpenAI Frontier Governance Framework
“"AI Safety Incident Response Plan (AIRP)" (§2.6). The FGF uses the terms "AI safety incident", "potential AI safety incident", "critical safety incident" and "serious incident" without defining them; for critical safety incidents it points to the TFAIA (California SB 53).”noneDU
confidence: medium
OpenAI Frontier Governance Framework
Google DeepMindShadow mapping
Google DeepMind Frontier Safety Framework v3.1
“"incidents relating to our frontier safety risk domains" (s.1.3.2); no definition.”noneDU
confidence: medium
Google DeepMind Frontier Safety Framework v3.1
xAIShadow mapping
xAI Frontier AI Framework, 30 June 2026
“"serious incident" (trigger point) and "Serious AI safety incidents: used, not defined" (s.2, s.3).”
The FAIF26 PDF's own metadata /Title reads "Privileged/Confidential DRAFT working FRAMEWORK DOC"; no xAI statement disambiguating draft vs. final status was found. Treat this row's provenance as a draft document, not a confirmed-final xAI policy, until that is resolved.
noneDU
confidence: medium
xAI Frontier AI Framework, 30 June 2026
MetaShadow mapping
Meta Advanced AI Scaling Framework v2
“"major incident" and "reporting critical incidents as appropriate" (§2.2.2, §2.3.2); undefined.”noneDU
confidence: medium
Meta Advanced AI Scaling Framework v2
EUShadow mapping
EU GPAI Code of Practice, Safety and Security Chapter
“"serious incident" is not given a one-sentence definition in the chapter, but Commitment 9 fully operationalises it: Signatories track, document and report "relevant information about serious incidents along the entire model lifecycle and possible corrective measures". Measure 9.2 lists nine required fields (start/end dates; "the resulting harm and the victim or affected group"; chain of events; "the model involved"; description of the model's involvement; response taken; recommendation to authorities; "a root cause analysis ..., including ... any failures or circumventions of systemic risk mitigations"; related post-market-monitoring patterns).”exactEQ
confidence: high
EU GPAI Code of Practice, Safety and Security Chapter
California SB 53Shadow mapping
California SB 53
“"Critical safety incident" means any of the following: (1) Unauthorized access to, modification of, or exfiltration of, the model weights of a frontier model that results in death or bodily injury. (2) Harm resulting from the materialization of a catastrophic risk. (3) Loss of control of a frontier model causing death or bodily injury. (4) A frontier model that uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk. (22757.11(d)).”exactEQ
confidence: high
California SB 53 (Transparency in Frontier Artificial Intelligence Act), Chapter 138, Statutes of 2025
UK AI Security Institute (AISI)Shadow mapping
UK AI Security Institute -- Incident Report: unsanctioned agent behaviour during cyber testing (4 Aug 2026)
“AISI published an incident report on its own cyber testing: "During a routine cyber evaluation, AISI identified an incident in which AI agents took sustained, unsanctioned action directed at real people and organisations." It "declared a security incident" and states: "We are treating this as a serious incident". The word "incident" is used without a definition.”
AISI uses "serious incident" in its ordinary sense; the post does not say whether the event met the EU or SB 53 legal definitions.
noneDU
confidence: medium
UK AI Security Institute -- Incident Report: unsanctioned agent behaviour during cyber testing (4 Aug 2026)
Frontier Model ForumShadow mapping
Frontier Model Forum -- Information Sharing, Incident Reporting and Incident Response Issue Brief
“"Incident" is not defined. "The definition and appropriate scope of what is considered an 'incident' depends on regulatory context, risk categories, and the capabilities of the systems involved."”noneDU
confidence: medium
Frontier Model Forum -- Information Sharing Issue Brief
EUShadow mapping
EU AI Act (Regulation (EU) 2024/1689), Article 3(49) and Article 55(1)(c)
“"'serious incident' means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person's health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment" (Art. 3(49)). Providers of general-purpose AI models with systemic risk must "keep track of, document, and report, without undue delay, to the AI Office and, as appropriate, to national competent authorities, relevant information about serious incidents and possible corrective measures to address them" (Art. 55(1)(c)).”
Article 3(49) is written for "an AI system"; the GPAI Code of Practice applies the same term to general-purpose AI models with systemic risk.
exactEQ
confidence: high
EU AI Act (Regulation (EU) 2024/1689), Article 3
OECDShadow mapping
OECD, Defining AI incidents and related terms (2024)
“"An AI incident is an event, circumstance or series of events where the development, use or malfunction of one or more AI systems directly or indirectly leads to any of the following harms: (a) injury or harm to the health of a person or groups of people; (b) disruption of the management and operation of critical infrastructure; (c) violations of human rights or a breach of obligations under the applicable law intended to protect fundamental, labour and intellectual property rights; (d) harm to property, communities or the environment." The paper also defines "serious AI incident", "AI hazard", "serious AI hazard" and "AI disaster".”
A definition offered for international interoperability, not a legal reporting obligation.
closeCL
confidence: high
OECD, Defining AI incidents and related terms (OECD Artificial Intelligence Papers No. 16, 2024)
US Congress (S. 5061, 119th Congress, not enacted)Shadow mapping
S. 5061, Sec. 2(3)
“"artificial intelligence safety incident" means "an event that materially increases the risk that operation of an artificial intelligence system leads to a state in which human life, health, property, or the environment is endangered." (Sec. 2(3))”
S. 5061 is a bill introduced in the US Senate on 21 July 2026; it has not been enacted.
closeCL
confidence: medium
S. 5061 (119th Congress)
What do these codes mean?
exact
The source term is equivalent to this element
close
The source term is close but not equivalent to this element
broad
The source term is broader than this element
narrow
The source term is narrower than this element
none
No mapping claim — used for false-friend and declared-but-undefined rows
EQ
Equivalent
CL
Close
BR
Source is broader than the element
NR
Source is narrower than the element
FF
False friend — same or similar label, different meaning
DU
Declared but undefined by the source
UV
Unverified

Related, not mapped

Pointers that are not crosswalk claims

These sources mention this concept but do not define or map it clearly enough to count as a crosswalk row — noted here so the research is visible without overstating it as a mapping.

Divergence

Where sources materially disagree

The word 'incident' collides across sources. California SB 53 and New York's RAISE Act give 'critical safety incident' a closed four-limb legal definition, the EU AI Act defines 'serious incident' by harm type, and the OECD offers a broader interoperable definition with separate tiers (AI incident, serious AI incident, AI disaster). OpenAI's Frontier Governance Framework, Google DeepMind's Frontier Safety Framework, xAI's framework, Meta's framework and UK AISI's incident report use the word without defining it, and the Frontier Model Forum states that the appropriate scope of 'incident' depends on regulatory context, risk categories and system capabilities. A record that says only 'incident' is therefore not comparable across sources unless it also names the definition it was assessed against.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →