Examples
Examples from published documents
- Anthropic's Level 2 and Level 3 classifiers "follow a two-stage design: some queries are flagged by a streaming classifier based on linear probes of model internals, which get passed on to a more expensive second stage classifier based on a fine-tuned model." (Anthropic Risk Report, August 2026, §4.5.1)
- Constitutional Classifiers are "safeguards trained on synthetic data, generated by prompting LLMs with natural language rules (i.e., a constitution) specifying permitted and restricted content." (Sharma et al., arXiv 2501.18837, abstract)
Counter-examples
Not to be confused with
- Refusal training changes the model itself; a classifier is a separate system wrapped around the model and can be updated without retraining it.
Related terms
Related AI safety terms
Terms defined under this element that are not NIKOLAI elements themselves, each as the cited document uses it. Every term is also listed in the AI safety glossary.
- Constitutional classifiers
- Constitutional classifiers are safeguard classifiers trained on synthetic data generated from a natural-language constitution that specifies permitted and restricted content. Anthropic introduced them to defend against universal jailbreaks.
- Source: Sharma et al., Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming (arXiv 2501.18837)
- Activation classifier
- An activation classifier inspects a model's internal activations during generation rather than its text. OpenAI describes pausing streaming to the user when the patterns suggest harmful content, while a separate check decides whether it is harmful.
- Source: GPT-5.6 deployment safety card
- Linear probe
- A linear probe is a lightweight classifier that reads a model's internal activations. Anthropic uses a streaming probe as the cheap first stage that flags queries for a more expensive second-stage classifier based on a fine-tuned model.
- Source: Anthropic Risk Report, August 2026
Source of record
Where this definition comes from
Anthropic Risk Report, August 2026, §4.5, footnote 60
“In general, our use of the word "classifier" in this report refers to the end-to-end system which makes a decision to flag or block part of a model exchange.”
https://www-cdn.anthropic.com/f61d49fa5596956a5dec75fea0e973bf6a6a8378/Redacted%20Risk%20Report%20August%202026%20.pdf
Crosswalk
How named organisations use this concept
Every row below is a shadow mapping. A shadow row is CASRAI's own reading of a published document. No lab, evaluator or regulator named on a shadow row has declared, endorsed, or been consulted on it. That changes only when an organisation files its own Mapping Declaration.
| Organisation | Their term, as published | Match & verification | Source |
|---|---|---|---|
| AnthropicShadow mapping Anthropic Risk Report, August 2026 | “"our use of the word 'classifier' in this report refers to the end-to-end system which makes a decision to flag or block part of a model exchange" (fn 60, inner quotation marks normalised); "some queries are flagged by a streaming classifier based on linear probes of model internals, which get passed on to a more expensive second stage classifier based on a fine-tuned model" (§4.5.1)” | exactEQ confidence: high | Anthropic Risk Report, August 2026 |
| OpenAIShadow mapping GPT-5.6 deployment safety card | “Two-tier system: "a fast, topical classifier model that determines whether or not the content is related to one of the two areas where we have deployed Preparedness Safeguards", then "a specially trained safety reasoner similar to gpt-oss-safeguard"; "activation classifiers monitor patterns in the model's internal activations"” | exactEQ confidence: high | GPT-5.6 deployment safety card |
| Google DeepMindShadow mapping Gemini 3.7 Flash FSF report | “Proactive misuse detection: relevant logs "pass through multiple levels of LLM-based classifiers to surface users attempting to elicit dangerous content"” | closeCL confidence: medium | Gemini 3.7 Flash FSF report |
| AmazonShadow mapping Amazon's Frontier Model Safety Framework | “"Input moderation systems" "detect and either block or safely modify prompts that contain malicious, insecure or illegal material, or attempt to bypass the core model alignment (e.g. prompt injection, jail-breaking)"; "Output moderation systems" block or safely modify violating outputs” | closeCL confidence: medium | Amazon's Frontier Model Safety Framework |
| G42Shadow mapping G42 Frontier Safety Framework | “Deployment Mitigation Level 2: "Real-Time Monitoring and Prompt Filtering: Real-time classifiers evaluate inputs and outputs, detecting ... and filtering harmful interactions as they occur."” The PDF's two-column table layout interleaves this sentence with the DML 2 objective; the ellipsis marks that break. | closeCL confidence: medium | G42 Frontier Safety Framework |
| xAIShadow mapping Frontier AI Framework, 30 Jun 2026 | “"Filters": "Applying classifiers to verify safety when a model is queried regarding topics of CRBN [sic] risks"” xAI's Frontier AI Framework (30 Jun 2026) carries PDF metadata reading "Privileged/Confidential DRAFT working FRAMEWORK DOC" with no xAI statement found disambiguating draft from final; the cited URL also returned HTTP 404 (NoSuchKey) on 13 Nov 2026, so the quote is verified against the local capture only. Treat as provisional. | closeCL confidence: medium | xAI Frontier AI Framework, 30 Jun 2026 |
| EUShadow mapping EU GPAI Code of Practice, Safety and Security Chapter | “Measure 5.1 example safety mitigation: "(2) monitoring and filtering the model's inputs and/or outputs"” | broadBR confidence: medium | EU GPAI Code of Practice, Safety and Security Chapter |
What do these codes mean?
- exact
- The source term is equivalent to this element
- close
- The source term is close but not equivalent to this element
- broad
- The source term is broader than this element
- narrow
- The source term is narrower than this element
- none
- No mapping claim — used for false-friend and declared-but-undefined rows
- EQ
- Equivalent
- CL
- Close
- BR
- Source is broader than the element
- NR
- Source is narrower than the element
- FF
- False friend — same or similar label, different meaning
- DU
- Declared but undefined by the source
- UV
- Unverified
Related, not mapped
Pointers that are not crosswalk claims
These sources mention this concept but do not define or map it clearly enough to count as a crosswalk row — noted here so the research is visible without overstating it as a mapping.
- Meta
LlamaFirewall, with "chain-of-thought auditing" and input classifiers, offered to the ecosystem (§4.2.1). Related, not a mapping.
Meta Advanced AI Scaling Framework v2
Gap
Few sources publish blocking thresholds or false-positive rates for deployed classifiers. Anthropic's Constitutional Classifiers paper reports "an absolute 0.38% increase in production-traffic refusals" for its research classifiers (arXiv 2501.18837).
