Skip to main content
v2026.11,986 entries · CC-BY 4.0
CASRAIAsk CASRAI · AI assistant for researchAsk a research question. Answers come with their sources.Funder rules, compliance, methods and publishing. Answers link the sources they rest on.
NIKOLAI elementN8 · Transparency and reviewProposednikolai-v0.3

Evaluator Independence and Conflict of Interest

Definition

Evaluator independence is the absence of financial, organisational or personal ties that could bias an outside evaluator's judgement of an AI developer, demonstrated through conflict-of-interest disclosures, recusals and funding terms that do not depend on the results.

Also called: Conflict of interest (evaluator), Independence of judgment, Methodological independence, Minimized Conflicts of Interest, Assessor independence

NIKOLAI's proposed operational definition

NIKOLAI proposal: Evaluator Independence and Conflict of Interest is a record of declared financial, organisational and personal relationships between an evaluator and the developer being evaluated, and the independence test applied to clear the evaluator for the engagement. This working definition is NIKOLAI's own editorial synthesis.

The operational definition is CASRAI's own proposal, not a definition any named organisation has agreed to. See what NIKOLAI is and is not, or browse the AI safety glossary.

Examples

Examples from published documents

  • Anthropic will not select an external reviewer whose organisation, reviewing staff or their managers hold a financial interest in Anthropic (Anthropic RSP v3.4, §3.6.1).
  • METR states that, per its standard policy, it took no payment from OpenAI for its incident investigation, while declaring the free API credits it accepted (METR, OpenAI-Hugging Face incident investigation).
  • A large frontier developer may not retain a compliance auditor if either party has a financial interest in the other (FRONTIER Act, H.R. 9925, Sec. 4(c), introduced, not enacted).

Counter-examples

Not to be confused with

  • Evaluator qualification (domain expertise and security protocols) is not independence: the EU Code of Practice's qualification test for external evaluators contains no explicit financial conflict-of-interest criterion.

Related terms

Related AI safety terms

Terms defined under this element that are not NIKOLAI elements themselves, each as the cited document uses it. Every term is also listed in the AI safety glossary.

Independent evaluator
An independent evaluator is an outside organisation assessing an AI developer's models or risk reports that has no financial interest in that developer and no major conflicts of interest among the people conducting the review.
Source: Anthropic Advanced AI Framework
Evaluator shopping
Evaluator shopping is the risk that AI developers choose whichever evaluators will ask the least of them and judge most generously; proposed remedies include government rating of evaluators and random assignment of highly rated ones.
Source: Anthropic Advanced AI Framework
Contingent compensation
Contingent compensation is payment to an evaluator that depends on the results of its evaluation. Independence standards for AI evaluations require that an evaluator's fee is not tied to whether its findings are favourable.
Source: AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025)
Recusal (evaluation)
Recusal in AI evaluation means excluding staff with a significant financial interest in the developer, such as direct equity, from defining or running the evaluation or approving its findings.
Source: AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025)
Conflict-of-interest disclosure
A conflict-of-interest disclosure is an evaluator's published statement of relationships that could bias an evaluation, such as payment from the developer or its competitors, so readers can weigh the findings accordingly.
Source: AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025)

Source of record

Where this definition comes from

Crosswalk

How named organisations use this concept

Every row below is a shadow mapping. A shadow row is CASRAI's own reading of a published document. No lab, evaluator or regulator named on a shadow row has declared, endorsed, or been consulted on it. That changes only when an organisation files its own Mapping Declaration.
OrganisationTheir term, as publishedMatch & verificationSource
AnthropicShadow mapping
Anthropic Responsible Scaling Policy v3.4 / Advanced AI Framework / We Must Pace the Frontier (essay by Anthropic's CEO)
“RSP v3.4 §3.6.1, three selection tests: reviewers "Have significant experience and expertise regarding evaluations for dangerous AI capabilities and propensities", including knowing "ways such evaluations might be misleading (for example, alignment faking)"; "Have reputational and other incentives making them likely to be candid" ("external review parties should not be teams whose revenue, reputation and success depend entirely on Anthropic and similar companies continuing to work with them"); and "Do not have conflicts of interest with respect to Anthropic. At a minimum, a reviewing organization itself may not have a financial interest in Anthropic; and the individuals involved in conducting the review, as well as anyone above them in the reporting chain within their organization, may not have a financial interest in Anthropic or close personal relationships with anyone at Anthropic." "In selecting external reviewers, we will consult with the Board and obtain the approval of the LTBT." AAF: "independent in the sense of not having a financial interest in the developer and being free of major conflicts of interest with respect to the individuals involved in conducting the review" (p.7); avoid "evaluator shopping." Essay: embedded evaluators "can simply provide a second opinion free of commercial incentives."”
Also cites the Advanced AI Framework and 'We Must Pace the Frontier'.
exactEQ
confidence: high
Anthropic Responsible Scaling Policy v3.4
EUShadow mapping
EU GPAI Code of Practice, Safety and Security Chapter
“Appendix 3.5 "adequate qualification" test: "(1) having significant domain expertise for the systemic risk and being technically skilled and experienced in conducting model evaluations; (2) having appropriate internal and external information security protocols in place; and (3) having agreed to protect commercially confidential information, if they need access to such information." No explicit financial-conflict-of-interest test (unlike Anthropic's RSP §3.6.1); Model Report must give "an explanation of the choice of evaluator based on the qualification criteria" (Measure 7.4).”
Notably lacks an explicit financial-COI test, unlike Anthropic's RSP.
closeCL
confidence: medium
EU GPAI Code of Practice, Safety and Security Chapter
METRShadow mapping
METR OpenAI-Hugging Face incident investigation / METR (About)
“"Per our standard policy, we did not take payment from OpenAI for this independent assessment"; "we accepted free API credits to conduct our experiments (also per our standard policy)", estimated at "roughly ~$400K in API credits". About page: "METR has not accepted funding from AI companies, though we make use of significant free tokens"; "We are committed to mitigating conflicts of interest where possible, and disclosing any unmitigated conflicts in our reports for these assessments."”
Also cites METR's About page (https://metr.org/about). Engagement-level disclosure: no payment, but in-kind API credits declared.
exactEQ
confidence: high
METR: OpenAI-Hugging Face incident investigation
Frontier Model ForumShadow mapping
FMF Third-Party Assessments technical report
“"Methodological independence enables assessors to approach evaluations with a fresh perspective" (5, robustness); open-ended assessment "can also involve internal assessors who are sufficiently independent to internal teams running capability assessment and mitigations" (3.1); effective assessments share "independence of judgment" (1.3).”
The report gives no financial, contractual or organisational test for independence.
closeCL
confidence: medium
FMF Third-Party Assessments
AI Evaluator Forum (AEF-1)Shadow mapping
AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations
“AEF-1 principle 2, "Minimized Conflicts of Interest", with conditions "Contingent compensation", "Organizational control", "CoI policy", "CoI disclosure", "Recusals" and "Separate agreements".”
Verified against the AEF-1 PDF (v1, 4 Dec 2025); see the other AEF-1 row for the verbatim condition texts.
closeCL
confidence: high
AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025)
US Congress (H.R. 9925, FRONTIER Act, not enacted)Shadow mapping
FRONTIER Act, 119th Congress
“Compliance audit: "A large frontier developer shall not retain a third party if either the developer or the third party has a financial interest in the other. A large frontier developer may compensate a third party for its services but shall not condition any payment, or the amount of any payment, on the results of the third party's audit"; the audit report lists "The third party's procedures for managing conflicts of interest and any conflicts of interest of personnel involved in the audit" (Sec. 4(c)). IVO licensing rules must include "Conflict-of-interest and funding-transparency requirements, including reporting requirements regarding the IVOs' funding sources and revenue generation and self-audit requirements regarding the IVOs' personnel and leadership to ensure adequate independence from the artificial intelligence industry" (Sec. 3(c)(1)). Each IVO report carries a certification that "the IVO is free from conflicts of interest" (Sec. 5(f)).”
Not enacted.
exactEQ
confidence: high
FRONTIER Act, 119th Congress (H.R. 9925, introduced, not enacted)
US Congress (S. 5061, 119th Congress, not enacted)Shadow mapping
S. 5061, 119th Congress
“The proposed AI Risk Board's conflict-of-interest policy: members have "a fiduciary responsibility to the Board, a duty to report conflicts of interest, including the appearance of a conflict of interest, and do not participate in deliberations or votes from which they personally or their employer would directly and materially benefit" (Sec. 3(b)(5)(D)(i)); the policy must "require each member to publicly disclose all relevant financial and employment relationships and include recusal procedures in the event of a conflict" (Sec. 3(b)(5)(D)(ii)); "Any member who fails to comply with the conflict of interest policy adopted pursuant to paragraph (5)(D) shall be removed from the Board" (Sec. 3(b)(3)(C)); a member denied a security clearance, or restricted from classified information, "shall be removed from the Board" (Sec. 3(b)(4)(A)(iv)).”
An institutional (board-member) COI regime rather than a reviewer-selection COI regime; in CASRAI's reading, a fuller independence regime than Anthropic's RSP §3.6.1 on institutional grounds. Not enacted.
closeCL
confidence: medium
S. 5061, 119th Congress
AI Evaluator Forum (AEF-1)Shadow mapping
AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations
“Principle 2 "Minimized Conflicts of Interest": "2.1 Contingent compensation Requirement: The evaluator did not receive compensation contingent on the results of the evaluation"; "2.2 Organizational control Requirement: The system provider did not exercise organizational or financial control over the evaluator"; "2.4 CoI disclosure Requirement: The evaluator clearly disclosed potential conflicts of interest relevant to the evaluation"; "2.5 Recusals Requirement: The evaluator recused any individuals with a significant financial interest in the system provider from carrying out the evaluation."”
Verified against the AEF-1 PDF (v1, 4 Dec 2025).
exactEQ
confidence: high
AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025)
What do these codes mean?
exact
The source term is equivalent to this element
close
The source term is close but not equivalent to this element
broad
The source term is broader than this element
narrow
The source term is narrower than this element
none
No mapping claim — used for false-friend and declared-but-undefined rows
EQ
Equivalent
CL
Close
BR
Source is broader than the element
NR
Source is narrower than the element
FF
False friend — same or similar label, different meaning
DU
Declared but undefined by the source
UV
Unverified

Related, not mapped

Pointers that are not crosswalk claims

These sources mention this concept but do not define or map it clearly enough to count as a crosswalk row — noted here so the research is visible without overstating it as a mapping.

  • OpenAI

    Proposal that CAISI certify third-party assessors: "independent assessment certification" (blueprint p.5) -- a pointer, not a mapping; this is an OpenAI proposal, not current practice at CAISSI (formerly CAISI).

    A Blueprint for a Federal Framework
  • Google DeepMind

    HuBREC: "a research ethics committee run within Google DeepMind but staffed and chaired by academics from outside the company" (p.21) -- a pointer, not a mapping.

    Gemini 3.7 Flash FSF report
  • Meta

    Evaluations "held out from teams conducting training to reduce the risk of overfitting" (§4.1) -- internal segregation; a pointer, not a mapping.

    Meta Advanced AI Scaling Framework v2
  • US Government

    Certification of third-party assessors by CAISSI (formerly CAISI) is an OpenAI proposal (see the OpenAI pointer above), not current practice -- a pointer, not a mapping.

    A Blueprint for a Federal Framework

Gap

California SB 53 and New York's RAISE Act (as amended by S8828) ask developers to describe third-party involvement but contain no independence or conflict-of-interest test for evaluators. The EU Code of Practice tests qualification and confidentiality rather than financial ties. No source defines a common disclosure format, so independence is currently stated engagement by engagement.

Referenced across the research world

University of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logoUniversity of Cambridge logoColumbia University logoCrossref logoUniversity of Edinburgh logoHarvard University logoUniversity of Oxford logoPrinceton University logoStanford School of Medicine logoUniversity College London logoORCID logo
  • University of Cambridge logo
  • Columbia University logo
  • Crossref logo
  • University of Edinburgh logo
  • Harvard University logo
  • University of Oxford logo
  • Princeton University logo
  • Stanford School of Medicine logo
  • University College London logo
  • ORCID logo

View CASRAI adoption →