Examples
Examples from published documents
- SecureBio, a nonprofit research organisation focused on catastrophic biological risk, evaluated two pre-release checkpoints of GPT-5.6 Sol and a version with biological content filters disabled (OpenAI GPT-5.6 System Card, s.9.1.1.8).
- The Model Report must explain the choice of external evaluator based on the qualification criteria (EU GPAI Code of Practice, Measure 7.4(3)).
Counter-examples
Not to be confused with
- A contractor hired to build or run the developer's own evaluations under the developer's direction is supplementing internal capacity, not acting as an independent evaluator in the AEF-1 sense.
- A regulator receiving a report is not thereby an evaluator; it becomes one only when it tests the model itself, as in government pre-deployment testing.
Related terms
Related AI safety terms
Terms defined under this element that are not NIKOLAI elements themselves, each as the cited document uses it. Every term is also listed in the AI safety glossary.
- Third-party evaluation
- Third-party evaluation is testing of an AI system by an organisation independent of its developer, valued for avoiding the conflict of interest that arises when providers evaluate their own systems.
- Source: AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025)
- Independent verification organization
- An independent verification organization (IVO) is, in the introduced US FRONTIER Act, a licensed private body that assesses a very large frontier developer's safety framework and risk mitigation at least every six months.
- Source: FRONTIER Act, 119th Congress (H.R. 9925, introduced, not enacted)
- AI Evaluator Forum
- The AI Evaluator Forum is a network of organisations that conduct independent, third-party evaluations of AI systems, sharing methods and aligning on professional norms; it publishes the AEF-1 standard.
- Source: AI Evaluator Forum (home page)
- AEF-1
- AEF-1 is a voluntary standard and checklist that third-party AI evaluators use to show the independence, access and transparency conditions under which an evaluation was carried out, published alongside their results.
- Source: AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025)
Source of record
Where this definition comes from
FMF Third-Party Assessments, Executive Summary; 1.1
“Third-party assessments can be conducted on frontier models to confirm evaluations or claims on critical safety capabilities and mitigations.”
https://www.frontiermodelforum.org/technical-reports/third-party-assessments/EU GPAI Code of Practice, Safety and Security Chapter, Appendix 3.5
“Adequate qualification of independent external evaluators requires: (1) having significant domain expertise for the systemic risk and being technically skilled and experienced in conducting model evaluations; (2) having appropriate internal and external information security protocols in place; and (3) having agreed to protect commercially confidential information, if they need access to such information.”
https://ec.europa.eu/newsroom/dae/redirection/document/118119California SB 53, 22757.12(a)(5)
“(5) Using third parties to assess the potential for catastrophic risks and the effectiveness of mitigations of catastrophic risks.”
https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260SB53
Crosswalk
How named organisations use this concept
| Organisation | Their term, as published | Match & verification | Source |
|---|---|---|---|
| Frontier Model ForumShadow mapping FMF Third-Party Assessments technical report | “"third-party assessors" performing three functions, "Confirmation", "Robustness" and "Supplementation"; open-ended assessment means "external parties applying their own evaluation methodologies" and "can also involve internal assessors who are sufficiently independent" (3.1); "third-party assessors must demonstrate their capacity to responsibly handle privileged access before receiving it" (5).” | exactEQ confidence: high | FMF Third-Party Assessments |
| EUShadow mapping EU GPAI Code of Practice, Safety and Security Chapter | “"independent external evaluators", who must meet the Appendix 3.5 "Adequate qualification" test (domain expertise and evaluation skill; information security protocols; agreement to protect commercially confidential information); Measure 7.4(3): the Model Report gives "an explanation of the choice of evaluator based on the qualification criteria."” | exactEQ confidence: high | EU GPAI Code of Practice, Safety and Security Chapter |
| AnthropicShadow mapping Anthropic Advanced AI Framework | “"qualified independent evaluator": Covered Developers should "regularly engage at least one qualified independent evaluator to examine the relevant covered models, gather information about safeguards, and provide their own assessment of Enumerated Risks" (p.7).” A legislative proposal, not a statement of current Anthropic practice. | exactEQ confidence: high | Anthropic Advanced AI Framework |
| California SB 53Shadow mapping California SB 53 | “"third parties" and "third-party evaluators" (undefined): frameworks describe "Using third parties to assess the potential for catastrophic risks and the effectiveness of mitigations of catastrophic risks" (22757.12(a)(5)); transparency reports summarise "The extent to which third-party evaluators were involved" (22757.12(c)(2)(C)). "Deploy" excludes "making a frontier model available to a third party for the primary purpose of developing or evaluating the frontier model" (22757.11(e)(2)).” The statute uses the term without defining it or setting qualification criteria. | closeCL confidence: medium | California SB 53 |
| OpenAIShadow mapping OpenAI Preparedness Framework v2 / GPT-5.6 System Card | “"then when available and feasible, OpenAI will work with third-parties to independently evaluate models" (PF §5.2). GPT-5.6: "SecureBio, a nonprofit research organization focused on catastrophic biological risk, evaluated two pre-release checkpoints of GPT-5.6 Sol and a railfree version of GPT-5.6 Sol" (s.9.1.1.8).” Also cites the GPT-5.6 deployment safety card. | closeCL confidence: medium | OpenAI Preparedness Framework v2 |
| xAIShadow mapping Grok 4.6 model card | “"We additionally provided an unrestricted configuration of Grok 4.6 to third-party evaluators, who corroborated the results of our internal evaluations and testing on cyber capabilities" (§7).” Evaluators are not named. | narrowNR confidence: medium | Grok 4.6 model card |
| AI Evaluator Forum (AEF-1)Shadow mapping AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations | “"independent third-party evaluations"; the standard "focuses specifically on evaluations where third parties maintain the freedom to define the evaluation's methodology", as opposed to "third parties simply validating, replicating, or supporting the execution of a provider's own evaluation" (Scope).” Verified against the AEF-1 PDF. Narrower than FMF's usage: AEF-1 excludes confirmation-only work from the independence level it covers. | closeCL confidence: high | AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025) |
| US Congress (H.R. 9925, FRONTIER Act, not enacted)Shadow mapping FRONTIER Act, 119th Congress | “"IVO" means "an independent verification organization operating in interstate or foreign commerce that is licensed under subsection (c) of section 5" (Sec. 2(14)).” A licensed sub-class of third-party evaluator. Not enacted. | narrowNR confidence: medium | FRONTIER Act, 119th Congress (H.R. 9925, introduced, not enacted) |
What do these codes mean?
- exact
- The source term is equivalent to this element
- close
- The source term is close but not equivalent to this element
- broad
- The source term is broader than this element
- narrow
- The source term is narrower than this element
- none
- No mapping claim — used for false-friend and declared-but-undefined rows
- EQ
- Equivalent
- CL
- Close
- BR
- Source is broader than the element
- NR
- Source is narrower than the element
- FF
- False friend — same or similar label, different meaning
- DU
- Declared but undefined by the source
- UV
- Unverified
Related, not mapped
Pointers that are not crosswalk claims
These sources mention this concept but do not define or map it clearly enough to count as a crosswalk row — noted here so the research is visible without overstating it as a mapping.
- US Government (NIST CAISI)
CAISSI (formerly CAISI) "will conduct pre-deployment evaluations and targeted research to better assess frontier AI capabilities and advance the state of AI security" under agreements with developers -- a government body in the evaluator role; a pointer, not a mapping.
NIST CAISI bulletin
Divergence
Where sources materially disagree
Sources split on whether a 'sufficiently independent' internal team can fill the role. The Frontier Model Forum allows it for open-ended assessment; the EU Code of Practice, Anthropic's Advanced AI Framework and AEF-1 require an external organisation. AEF-1 further excludes evaluators that only replicate or validate a developer's own method.
Gap
No public register or accreditation body for third-party evaluators exists in the sources. The Frontier Model Forum lists 'accreditation frameworks to identify qualified assessors' as future work; the EU Code of Practice names no accreditor; SB 53 does not define 'third-party evaluator'.
