Skip to main content
v2026.11,986 entries · CC-BY 4.0
CASRAIAsk CASRAI · AI assistant for researchAsk a research question. Answers come with their sources.Funder rules, compliance, methods and publishing. Answers link the sources they rest on.
NIKOLAI elementN8 · Transparency and reviewProposednikolai-v0.3

External Review

Definition

External review is an assessment of an AI model, a risk report, a safeguard or a developer's compliance with its own safety framework, carried out by a party outside the developer, whose findings are recorded with the review's type, scope and output.

Also called: Third-party assessment, External evaluation, Independent external evaluation, External safety testing, Third-party review

NIKOLAI's proposed operational definition

NIKOLAI proposal: an External Review is a record of an assessment performed by a party outside the model developer -- of a model, a risk report, a safeguard, or compliance with a framework -- capturing the review's type, scope, and output. This working definition and the record shape below are NIKOLAI's own editorial synthesis; no lab, evaluator or regulator has endorsed it.

The operational definition is CASRAI's own proposal, not a definition any named organisation has agreed to. See what NIKOLAI is and is not, or browse the AI safety glossary.

Examples

Examples from published documents

  • Anthropic's policy commits to sharing a Risk Report with external reviewers within one week of submitting it to its Board and Long-Term Benefit Trust, and asking them for public commentary within 30 days of receipt (Anthropic RSP v3.4, §3.6.2).
  • OpenAI gave the UK AI Security Institute early access to GPT-5.6 Sol to support independent, pre-deployment evaluation of cyber capabilities (OpenAI GPT-5.6 System Card, s.9.1.2.6).
  • Signatories must have adequately qualified independent external evaluators run model evaluations unless the model is 'similarly safe or safer' or no qualified evaluator can be found after a public call (EU GPAI Code of Practice, Appendix 3.5).

Counter-examples

Not to be confused with

  • Internal review by a team inside the developer is not external review, even when it is organisationally separate from the training team; the Frontier Model Forum accepts 'sufficiently independent' internal assessors only for open-ended assessment.
  • A developer's own summary of an outside evaluator's findings is not the external review itself; the record should point to the evaluator's own report where one exists.

Related terms

Related AI safety terms

Terms defined under this element that are not NIKOLAI elements themselves, each as the cited document uses it. Every term is also listed in the AI safety glossary.

Third-party assessment
Third-party assessment is the industry term for an evaluation of a frontier model by an outside party to confirm developer evaluations or safety claims, test them with independent methods, or add specialist expertise the developer lacks.
Source: FMF Third-Party Assessments
Confirmatory assessment
A confirmatory assessment is a third-party review that checks how a developer ran its own evaluations and whether its conclusions are supported, rather than applying new methods. Replication testing, methodology review and holistic review are its usual forms.
Source: FMF Third-Party Assessments
Government testing
Government testing is evaluation of AI systems by a public testing authority, such as a national AI institute, which governments in the United States, United Kingdom, European Union and China rely on as a core part of their approach to AI risk.
Source: AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025)

Source of record

Where this definition comes from

  • Anthropic Risk Report, August 2026, §1.3.5

    “Version 3.2 of our RSP authorizes Anthropic's Long-Term Benefit Trust (LTBT) to request external review of risk reports, gives the LTBT the power to approve our selection of external reviewers ... every part of any unredacted risk report is reviewed by at least one such reviewer.”

    https://www-cdn.anthropic.com/f61d49fa5596956a5dec75fea0e973bf6a6a8378/Redacted%20Risk%20Report%20August%202026%20.pdf
  • Anthropic Responsible Scaling Policy v3.4, §3.6

    “We will work toward a practice of seeking comprehensive, public external review on our Risk Reports. This means working with one or more third-party organizations that will receive private versions of our Risk Reports (unredacted or with minimal redactions, as discussed below) and publish comprehensive commentary on them.”

    https://cdn.sanity.io/files/4zrzovbb/website/0bacdc8440ea96e62a8766d99ebe1d4eea6d5f3a.pdf
  • EU GPAI Code of Practice, Safety and Security Chapter, Appendix 3.5

    “In addition to internal model evaluations, Signatories will ensure that adequately qualified independent external evaluators conduct model evaluations pursuant to this Appendix 3, with regards to the systemic risk, unless: (1) the model is a similarly safe or safer model pursuant to Appendix 2.2; or (2) Signatories fail to appoint adequately qualified independent external evaluators, despite using early search efforts (such as through a public call open for 20 business days) and promptly notifying identified evaluators”

    https://ec.europa.eu/newsroom/dae/redirection/document/118119
  • METR, accountability element

    “External scrutiny ensures that a company's safety claims can be independently validated by qualified experts.”

    https://metr.org/

Crosswalk

How named organisations use this concept

Every row below is a shadow mapping. A shadow row is CASRAI's own reading of a published document. No lab, evaluator or regulator named on a shadow row has declared, endorsed, or been consulted on it. That changes only when an organisation files its own Mapping Declaration.
OrganisationTheir term, as publishedMatch & verificationSource
AnthropicShadow mapping
Anthropic Risk Report, August 2026
“"Version 3.2 of our RSP authorizes Anthropic's Long-Term Benefit Trust (LTBT) to request external review of risk reports, gives the LTBT the power to approve our selection of external reviewers" ... "so long as every part of any unredacted risk report is reviewed by at least one such reviewer." "Since this change to our RSP, the LTBT has not requested an external review (nor has the RSP required that we conduct one), though we have continued to conduct pilot external reviews" (§1.3.5).”
Also draws on the Advanced AI Framework's evaluator-review contents (AAF p.8).
exactEQ
confidence: high
Anthropic Risk Report, August 2026
OpenAIShadow mapping
OpenAI Preparedness Framework v2 / Frontier Governance Framework
“"then when available and feasible, OpenAI will work with third-parties to independently evaluate models" (PF §5.2); "We may solicit and obtain input from external experts" (FGF §5).”
Also cites Frontier Governance Framework §5.
closeCL
confidence: medium
OpenAI Preparedness Framework v2
Google DeepMindShadow mapping
Gemini 3.7 Flash FSF report
“"External safety testing" by "specialist independent groups"; "The independent external evaluators explore frontier safety risk domains, however they do not directly comment on risk thresholds as contemplated under our FSF." (p.6)”closeCL
confidence: medium
Gemini 3.7 Flash FSF report
EUShadow mapping
EU GPAI Code of Practice, Safety and Security Chapter
“"Signatories will ensure that adequately qualified independent external evaluators conduct model evaluations pursuant to this Appendix 3, with regards to the systemic risk", unless exempt as 'similarly safe or safer' or unable to find qualified evaluators 'despite using early search efforts.' Model Report must explain 'the choice of evaluator based on the qualification criteria' and whether independent evaluator input 'informed' the decision to proceed (Measures 7.2, 7.4).”exactEQ
confidence: high
EU GPAI Code of Practice, Safety and Security Chapter
California SB 53Shadow mapping
California SB 53
“Framework topic "(5) Using third parties to assess the potential for catastrophic risks and the effectiveness of mitigations of catastrophic risks"; transparency report summary of "(C) The extent to which third-party evaluators were involved."”closeCL
confidence: medium
California SB 53
US Government (NIST CAISI) / UK AI Security Institute (AISI) partnershipShadow mapping
NIST CAISI bulletin; DeepMind UK AISI partnership post
“CAISI: "CAISI will conduct pre-deployment evaluations and targeted research to better assess frontier AI capabilities." UK AISI with DeepMind: "partnered with the UK AISI since its inception in November 2023 to test our most capable models."”
Also cites Google DeepMind's post on its partnership with the UK AI Security Institute.
closeCL
confidence: medium
NIST CAISI bulletin
METRShadow mapping
METR (site)
“"External scrutiny ensures that a company's safety claims can be independently validated by qualified experts."”exactEQ
confidence: high
METR
Frontier Model ForumShadow mapping
FMF Third-Party Assessments technical report
“"Third-party assessment: 'Third-party assessments can be conducted on frontier models to confirm evaluations or claims on critical safety capabilities and mitigations.'" Functions: "Confirmation", "Robustness", "Supplementation".”exactEQ
confidence: high
FMF Third-Party Assessments
G42Shadow mapping
G42 Frontier Safety Framework
“"G42 will engage in annual external audits to verify compliance with the Framework" (s.5).”closeCL
confidence: medium
G42 Frontier Safety Framework
AnthropicShadow mapping
Policy on the AI Exponential (essay by Anthropic's CEO)
“"Models above a threshold of compute should undergo mandatory testing by a qualified third party for their level of risk in four specific areas: cybersecurity, biological weapons, loss of control of AI systems, and automated R&D." Delivery left as a choice between "a government agency (similar to the FAA)" or "a set of private organizations that are authorized and inspected by the government ... (a 'regulatory markets' approach)."”
Proposes a distinct, two-track mandatory external-review architecture separate from the voluntary/embedded-evaluator model in 'We Must Pace the Frontier'; neither track specifies evaluator access level, publication rights or independence criteria -- a gap the essay itself does not fill.
closeCL
confidence: medium
Policy on the AI Exponential (essay by Anthropic's CEO)
US Congress (H.R. 9925, FRONTIER Act, not enacted)Shadow mapping
FRONTIER Act, 119th Congress
“"A very large frontier developer shall grant an IVO timely access upon request to unredacted materials, records, personnel, systems, and all other information reasonably necessary for conducting the assessments and ongoing monitoring required by this section"; "the IVO shall prepare and submit an assessment report not less frequently than once every 6 months" (Sec. 5). Separately, "a large frontier developer shall retain a third party to perform an independent audit of compliance with the frontier AI framework", annually, with "the signature of the lead auditor certifying the results of the audit" (Sec. 4(c)).”
IVO access is 'upon request', not standing employee-like access, and the bill does not set publication rights for assessors beyond the redacted public copy. Not enacted.
closeCL
confidence: medium
FRONTIER Act, 119th Congress (H.R. 9925, introduced, not enacted)
AnthropicShadow mapping
Anthropic Responsible Scaling Policy v3.4
“"We will work toward a practice of seeking comprehensive, public external review on our Risk Reports"; reviewers "publish comprehensive commentary" on "the quality of our reasoning, the validity of our risk assessments, the overall level of risk, and whether the redactions we've made for the public version are reasonable and appropriate"; a full external review is required "anytime a Risk Report covers highly capable models and is significantly redacted" (§3.6); "We will ask the external reviewers to provide public commentary on our report within 30 days of receipt" (§3.6.2).”
Policy text; the Risk Report of August 2026 (§1.3.5) records that the LTBT had not yet requested a review and that reviews were still at pilot stage.
exactEQ
confidence: high
Anthropic Responsible Scaling Policy v3.4
Frontier Model ForumShadow mapping
FMF Third-Party Assessments technical report
“"Third-party assessments can complement – but do not replace – internal safety processes and corporate governance mechanisms" (1.1); shared characteristics of effective assessments: "independence of judgment, technical rigor, clear methodology, appropriate expertise for the risks being evaluated, and actionable recommendations" (1.3).”
Describes member-firm practice 'where appropriate'; no obligation to engage.
closeCL
confidence: medium
FMF Third-Party Assessments
What do these codes mean?
exact
The source term is equivalent to this element
close
The source term is close but not equivalent to this element
broad
The source term is broader than this element
narrow
The source term is narrower than this element
none
No mapping claim — used for false-friend and declared-but-undefined rows
EQ
Equivalent
CL
Close
BR
Source is broader than the element
NR
Source is narrower than the element
FF
False friend — same or similar label, different meaning
DU
Declared but undefined by the source
UV
Unverified

Related, not mapped

Pointers that are not crosswalk claims

These sources mention this concept but do not define or map it clearly enough to count as a crosswalk row — noted here so the research is visible without overstating it as a mapping.

  • xAI

    CASRAI's reading of the June 2026 framework is that it contains no commitment to external or third-party evaluation. Cards separately state "third-party testing shows that Grok 4's end-to-end offensive cyber capabilities remain below the level of a human professional" and that third-party evaluators "corroborated" Grok 4.6 cyber results -- a pointer, not a mapping.

    xAI Frontier AI Framework, 30 Jun 2026
  • Meta

    External experts used in threat-modeling workshops, risk assessments "where appropriate" and red teaming "when appropriate" -- a pointer, not a mapping.

    Meta Advanced AI Scaling Framework v2

Gap

California SB 53 requires a large frontier developer's framework to describe how it uses third parties to assess catastrophic risk, and its transparency report to summarise "the extent to which third-party evaluators were involved", but the statute sets no independence, qualification or access standard for those evaluators. The Frontier Model Forum report describes practice without a disclosure or publication-rights regime. Only the EU Code of Practice and the introduced FRONTIER Act tie external evaluation to stated qualification criteria.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →