Who Checks the Checkers: Evaluator Independence in AI Safety

Frontier AI labs increasingly submit their models to outside evaluators before or after release, and every one of those arrangements rests on a single, mostly unexamined word: independent. Who counts as independent, and how that’s checked, varies enormously across the organizations doing the checking — from a named, testable financial-interest rule at one AI lab, to a general “adequate qualification” standard in EU policy that never mentions money at all, to a nonprofit evaluator that discloses its own funding refusals in public, to two competing bills in the US Congress that would regulate evaluator independence in structurally different ways. This guide walks through what each of those approaches actually commits to, and where CASRAI’s own NIKOLAI reference project has mapped — and flagged gaps in — each one.

The clearest test on record: Anthropic’s financial-interest and personal-relationship rule

Anthropic’s Responsible Scaling Policy (RSP), currently version 3.4, is the most specific public independence test any frontier lab has written down for its own external reviewers. Section 3.6.1, “Selecting external reviewers,” states that Anthropic will select reviewers who:

“Do not have conflicts of interest with respect to Anthropic. At a minimum, a reviewing organization itself may not have a financial interest in Anthropic; and the individuals involved in conducting the review, as well as anyone above them in the reporting chain within their organization, may not have a financial interest in Anthropic or close personal relationships with anyone at Anthropic (i.e., family relationships, romantic relationships, or shared living arrangements).”

Three things make this test unusually concrete compared to most public independence language: it names a specific prohibited relationship type (financial interest), it reaches beyond the individual reviewer to “anyone above them in the reporting chain,” and it defines “close personal relationship” by example rather than leaving it as an undefined term. It applies specifically to the external review of Anthropic’s own Risk Reports under RSP Section 3.6 — a narrower scope than “every third-party evaluation of an Anthropic model,” but a real, binding commitment within that scope, not aspirational language.

METR: refusing compensation, and disclosing conflicts rather than just avoiding them

METR, the independent nonprofit evaluator, takes a disclosure-and-recusal approach rather than a single bright-line rule. Its public conflict-of-interest policy states plainly on its risk-assessment page: “METR does not accept compensation for this work. Companies such as Anthropic, OpenAI, and xAI have provided access and tokens for our evaluations, research, and engineering” — drawing a specific line between accepting non-cash research access (which it does) and accepting payment for its assessments (which it says it does not).

METR’s published conflict-of-interest policy goes further than a blanket statement, setting out tiers of conflict for its own staff with different disclosure consequences. A current romantic relationship with an employee of the company being assessed, or living with one, sits at its more serious tier and generally requires disclosure in the relevant public report unless an unconflicted staff member independently checks and takes responsibility for that person’s contribution. Financial ties — equity, debt, current or recently-pursued paid work with the company — sit at the tier requiring disclosure without exception. The policy’s own framing is direct about the stakes: “It is our highest priority that our assessments of companies are accurate and impartial. We are committed to mitigating conflicts of interest where possible, and disclosing any unmitigated conflicts in our reports.”

That combination — a public compensation refusal plus a named, tiered disclosure framework covering personal and financial relationships — is one of the more concrete real-world implementations of “evaluator independence” that any organization in this space has put in writing, and it maps closely to what CASRAI’s NIKOLAI project treats as the core content of the element (below).

The EU’s GPAI Code: qualification, not conflict-of-interest

The EU AI Act’s General-Purpose AI Code of Practice takes a noticeably different approach for the independent external evaluators it requires signatories to use for systemic-risk model evaluations. Appendix 3.5 of the Code’s Safety and Security chapter defines “adequate qualification” for these evaluators as requiring only:

“(1) having significant domain expertise for the systemic risk and being technically skilled and experienced in conducting model evaluations; (2) having appropriate internal and external information security protocols in place; and (3) having agreed to protect commercially confidential information, if they need access to such information.”

Notice what is absent from that list: there is no financial-interest test, no personal-relationship test, and no requirement that the evaluator be free of ties to the developer whose model it is assessing at all. The Code does require signatories to publish “suitable criteria for assessing applications” when selecting evaluators, and to explain the choice of evaluator “based on the qualification criteria” in their Model Report — but the qualification criteria the Code itself sets are about competence and confidentiality, not independence in the conflict-of-interest sense that Anthropic’s RSP or METR’s own policy addresses. For an EU-facing compliance program, that gap matters: passing the Code’s own bar for an “adequately qualified” evaluator does not, by itself, establish that the evaluator has no financial or organizational stake in a favorable result.

Frontier Model Forum: independence criteria not yet defined

The Frontier Model Forum‘s public technical report on frontier capability assessments describes an internal-organizational-separation practice — “most frontier model developers maintain assessment teams that are separate from model development teams” — and notes that developers may bring in external red-teamers or domain specialists to assist. What the report does not do is set out criteria for when an external evaluator counts as independent: it specifies no financial, contractual, or organizational conflict-of-interest test for those external experts. That absence is not something the Forum’s own report calls out as a limitation in its own text; it is an observation from reading what the document does and does not commit to, which is worth being explicit about given how easy it is to conflate “a document doesn’t mention X” with “the document itself admits it doesn’t address X.”

Two competing, not-yet-enacted bills in the US Congress

Two 2026 bills before the US Congress would regulate evaluator independence directly, and they take structurally different approaches from each other — and from every industry or EU approach above. Neither has been enacted; both are introduced legislation only.

The FRONTIER Act (H.R. 9925, formally the Frontier Risk Oversight, National Transparency, Independent Evaluation, and Reporting Act) would license Independent Verification Organizations (IVOs) directly. It requires “conflict-of-interest and funding-transparency requirements, including reporting requirements regarding the IVOs’ funding sources and revenue generation and self-audit requirements regarding the IVOs’ personnel and leadership to ensure adequate independence from the artificial intelligence industry” (Sec. 3(c)(1)), conditions licensure on a finding that “the applicant has demonstrated its independence from the artificial intelligence industry” (Sec. 3(c)(2)(B)(i)), and requires each assessment report to carry the IVO’s lead assessor personally certifying “that the IVO is free from conflicts of interest, as defined by regulations issued under this Act” (Sec. 5(f)(2)(F)(iii)). Like Anthropic’s RSP, the FRONTIER Act’s independence test is ultimately anchored to individuals — a named lead assessor signs a certification — but it layers a licensure and funding-transparency regime around that individual attestation that RSP does not attempt.

The Secure Artificial Intelligence Development Act of 2026 (S. 5061, introduced by Sen. Mark Warner) takes a different structural approach entirely: rather than regulating evaluator organizations, it creates a governmental Artificial Intelligence Risk Board whose independence is enforced at the board-member level. Under Sec. 3(b)(5)(D), board members carry “a fiduciary responsibility to the Board, a duty to report conflicts of interest,” must recuse from deliberations where they or their employer would “directly and materially benefit,” and must “publicly disclose all relevant financial and employment relationships.” Board membership itself is required to include independent technical experts “not affiliated with a developer or provider of artificial intelligence systems” (Sec. 3(b)(2)(B)(i)). Where the FRONTIER Act licenses private evaluator organizations and RSP vets external reviewers case by case, S. 5061 builds independence into the composition and fiduciary obligations of a standing government body — a genuinely different model of where the independence guarantee is supposed to live.

One further reference point is worth naming honestly rather than treating as settled: an “AI Evaluator Forum” surfaces in discovery-stage materials with language resembling “minimized conflicts of interest,” but this could not be verified against a stable, citable primary source in the way the organizations above could. It is flagged here as unverified rather than presented as a data point.

What NIKOLAI maps, and what it doesn’t

CASRAI’s own NIKOLAI project — an independent, unendorsed reference dictionary for frontier AI safety, not a standard any lab or regulator has agreed to — formalizes this comparison as a named element: Evaluator Independence and Conflict of Interest, part of Track N8, Transparency and Review. NIKOLAI defines the element as a record documenting “declared financial, organisational and personal relationships between an evaluator and the developer being evaluated, and the independence test applied to clear the evaluator for the engagement” — and its own crosswalk table carries seven rows against that definition: Anthropic, the EU, METR, the Frontier Model Forum, the AI Evaluator Forum, H.R. 9925, and S. 5061.

It matters which of those rows reflect an organization’s own explicit declaration and which reflect NIKOLAI’s own reading of a public document. NIKOLAI’s page is explicit about this: “This is CASRAI’s own proposed definition, not a definition any named organisation has agreed to,” and every row in the crosswalk is what NIKOLAI calls a shadow mapping — CASRAI’s independent interpretation of what a published document says against NIKOLAI’s own element definition, not a confirmation from the mapped organization that this is how it describes itself. Anthropic did not write its RSP against NIKOLAI’s schema; METR did not draft its conflict-of-interest policy to match an N8.1 field; the bill text of H.R. 9925 and S. 5061 exists independently of NIKOLAI entirely. NIKOLAI’s contribution is reading each of those real, independently-published documents side by side against one consistent question — and being transparent that doing so is an act of interpretation, not certification. NIKOLAI does not certify that Anthropic, METR, or any other organization is actually independent in practice; it only records what each one has published about how it tries to be.

For the fuller landscape of how third-party evaluation works across embedded and arms-length engagements — access agreements, publication rights, and evaluation-validity questions beyond independence — see CASRAI’s guide to third-party AI evaluator standards and methodology, which sits a layer above this guide’s institution-by-institution detail. For the full N1–N10 track structure NIKOLAI uses to organize this material, see NIKOLAI’s track system, and for what NIKOLAI is and isn’t, see What Is NIKOLAI?

Frequently asked questions

What does Anthropic’s RSP require for external reviewer independence?

Anthropic’s RSP v3.4, Section 3.6.1, requires that external reviewers of its Risk Reports have no financial interest in Anthropic (at both the organizational and individual-reviewer level, including anyone above the reviewer in their reporting chain) and no close personal relationships — defined as family relationships, romantic relationships, or shared living arrangements — with anyone at Anthropic.

Does the EU’s GPAI Code require evaluators to be financially independent of the developer?

Not explicitly. The Code’s Appendix 3.5 defines “adequate qualification” for independent external evaluators in terms of domain expertise, technical skill, information security protocols, and confidentiality agreements — it does not set out a financial or organizational conflict-of-interest test the way Anthropic’s RSP or METR’s own policy does.

Has the FRONTIER Act or S. 5061 been enacted?

No. As of this writing, both H.R. 9925 (the FRONTIER Act) and S. 5061 (the Secure Artificial Intelligence Development Act of 2026) are introduced bills in the 119th Congress, not enacted law.

What’s the difference between the FRONTIER Act and S. 5061 on independence?

The FRONTIER Act would license private Independent Verification Organizations directly, conditioning licensure on demonstrated independence and funding transparency, with a named lead assessor certifying freedom from conflicts on each report. S. 5061 instead builds independence into a standing government Artificial Intelligence Risk Board, requiring board members to carry fiduciary duties, disclose financial and employment relationships, and recuse from matters where they’d materially benefit — independence enforced through board composition and governance rather than through licensing outside evaluator organizations.

Is NIKOLAI’s evaluator-independence crosswalk an official standard?

No. NIKOLAI is CASRAI’s own, unendorsed reference project. Its crosswalk rows are shadow mappings — CASRAI’s own reading of each organization’s published documents against NIKOLAI’s element definition — not declarations the mapped organizations have made about themselves, and not a certification that any evaluator is independent in practice.