Third-Party Evaluation & Assurance
The independent evaluation ecosystem around frontier AI: METR, the UK AI Security Institute, the US CAISI, embedded and arms-length evaluator models, evaluator independence standards, and third-party AI auditing.
Guides
External Review, Unpacked: Which Frontier Labs Let Outsiders Audit Their Safety Cases
Anthropic’s Long-Term Benefit Trust can compel outside review of its Risk Reports. California now requires labs to disclose how much third-party evaluation they used. Most other frontier labs’ safety policies still leave external review optional, or don’t mention it at all. This guide compares, lab by lab, who actually lets outsiders check their Responsible Scaling Policy compliance and who is still grading its own homework.
Apollo Research: AI Scheming Detection Explained
An institutional profile of Apollo Research: the AI safety lab behind scheming and deceptive-alignment research, its OpenAI evaluation collaboration, the Watcher runtime monitor, and its January 2026 move to an independent Public Benefit Corporation.
Is AI Red-Teaming Security Theater? What the Research Says
Carnegie Mellon researchers surveyed the AI red-teaming literature and analyzed six real exercises — Bing Chat, GPT-4, Gopher, two Claude releases, and DEF CON — and found practitioner definitions diverge so widely that treating red-teaming as a catch-all safety guarantee “verges on security theater.” Their answer isn’t to abandon the practice; it’s a 3-phase question bank for doing it rigorously.
The Access Record No Lab Currently Publishes
NIKOLAI’s Evaluator Access Attestation element says so itself: it is CASRAI’s own editorial synthesis, unsourced from any single document, because no developer currently publishes a record in this shape. This page assembles the eight fragments that exist — and finds the one place a lab volunteers to go further than the rest.
Building a Registry That Can’t Accidentally Lie
A design case study of NIKOLAI’s own Mapping Declarations mechanism (Track N10) for people building other open registries with organizational-attribution claims — the honesty gradient across six roles, a same-day dispute-bug fix, and why the roles CRediT doesn’t have were the ones worth building.
When AI Reviews AI: Inside NIKOLAI’s AI-Model-Review Element
Anthropic’s August 2026 Risk Report names the exact model instance that reviewed a draft of its own alignment section — Claude Mythos 5, with named internal access, in 24 minutes. That is not human-evaluator independence; it’s an AI system acting as an assurance actor. NIKOLAI’s AI-Model Review element, in CASRAI’s own independent dictionary, is built to record exactly that — and to name the human still accountable for what the AI found.
What Anthropic Pays for a Jailbreak: Inside Its Model Safety Bug Bounty
Anthropic will pay up to $15,000 through HackerOne for a novel, universal jailbreak against its CBRN and cybersecurity safeguards. Here’s exactly what that program covers, what it doesn’t, and where it sits inside Anthropic’s own safety commitments.
Evaluation-Validity Threats: Sandbagging, Reward Hacking, and NIKOLAI’s N5 Crosswalk
NIKOLAI’s N5 element defines the conditions — sandbagging, alignment faking, metagaming, reward hacking — where a model’s behavior in evaluation diverges from its behavior in deployment. This is its first per-element deep dive: the 8-org crosswalk, the sandbagging/alignment-faking distinction, Anthropic’s reward-hacking data, SB 53’s elicitation carve-out, and a live Senate investigation as case study.
Who Checks the Checkers: Evaluator Independence in AI Safety
Anthropic’s RSP names a financial-interest and personal-relationship test for its external reviewers; the EU’s GPAI Code asks only for “adequate qualification”; METR discloses its own conflicts and refuses compensation; and two competing, not-yet-enacted US bills would regulate evaluator independence in structurally different ways — an evaluator-by-evaluator look at what “independent” is actually required to mean, mapped against CASRAI’s own NIKOLAI crosswalk.
Mapping Declarations: How Organizations Verify and Confirm Their Own AI Safety Terminology in NIKOLAI
Every NIKOLAI crosswalk row starts as an unconfirmed shadow mapping. Here’s how an organization verifies its identity and gets a reviewed declaration live.
The International AI Safety Institute Network: What It Is, Now NAAIMES
There’s a coordination body connecting the individual national AI safety institutes — CAISI, UK AISI and others — to each other. It was founded in November 2024, and has since been renamed NAAIMES. Here’s what it actually does.
The Frontier Model Forum: What It Is and What It Does
An institutional profile of the Frontier Model Forum (FMF): its founding members, mission, funding, and how it differs from independent evaluators like METR and government bodies like UK AISI and US CAISI.
UK AI Safety Institute vs AI Security Institute: The 2025 Rename Explained
The UK government renamed its AI Safety Institute to the AI Security Institute on 14 February 2025. It is the same organisation under a new name, not two separate bodies — this guide explains what changed, why, and why the old name still turns up.
Third-Party AI Evaluator Standards: Independence, Access, and Methodology
A cross-cutting guide to the standards questions that apply to every third-party AI evaluator: embedded vs. arms-length access, independence and conflict-of-interest, evaluation validity, red-team access agreements, publication rights, and retaliation protection — mapped to NIKOLAI’s N8 Transparency and Review track.
UK AI Security Institute’s Frontier AI Trends Report: What Its Evaluations Have Found
AISI’s first Frontier AI Trends Report aggregates two years of evaluations across 30+ frontier models. Here is what it found on cyber, bio/chem, autonomy, and safeguards.
What Is METR? How Its AI Safety Evaluations Work
An institutional profile of METR (Model Evaluation and Threat Research): what it is, what it evaluates, how its evaluations work, and its relationship to the AI labs it assesses.
Third-Party AI Auditing: What It Is and Who Does It
Third-party AI auditing means an independent party assessing an AI system, or the organization running it, against a compliance framework like ISO/IEC 42001 or the EU AI Act — not the same as using AI tools to automate financial audits.
CAISI and the UK AI Security Institute: How Pre-Deployment Testing Agreements Work
CAISI and the UK AI Security Institute both renamed themselves in 2025 and both run voluntary pre-deployment testing agreements with frontier AI labs. Here’s what those agreements actually cover, and what “early access” means in practice.
