Examples
Examples from published documents
- Anthropic will not select an external reviewer whose organisation, reviewing staff or their managers hold a financial interest in Anthropic (Anthropic RSP v3.4, §3.6.1).
- METR states that, per its standard policy, it took no payment from OpenAI for its incident investigation, while declaring the free API credits it accepted (METR, OpenAI-Hugging Face incident investigation).
- A large frontier developer may not retain a compliance auditor if either party has a financial interest in the other (FRONTIER Act, H.R. 9925, Sec. 4(c), introduced, not enacted).
Counter-examples
Not to be confused with
- Evaluator qualification (domain expertise and security protocols) is not independence: the EU Code of Practice's qualification test for external evaluators contains no explicit financial conflict-of-interest criterion.
Related terms
Related AI safety terms
Terms defined under this element that are not NIKOLAI elements themselves, each as the cited document uses it. Every term is also listed in the AI safety glossary.
- Independent evaluator
- An independent evaluator is an outside organisation assessing an AI developer's models or risk reports that has no financial interest in that developer and no major conflicts of interest among the people conducting the review.
- Source: Anthropic Advanced AI Framework
- Evaluator shopping
- Evaluator shopping is the risk that AI developers choose whichever evaluators will ask the least of them and judge most generously; proposed remedies include government rating of evaluators and random assignment of highly rated ones.
- Source: Anthropic Advanced AI Framework
- Contingent compensation
- Contingent compensation is payment to an evaluator that depends on the results of its evaluation. Independence standards for AI evaluations require that an evaluator's fee is not tied to whether its findings are favourable.
- Source: AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025)
- Recusal (evaluation)
- Recusal in AI evaluation means excluding staff with a significant financial interest in the developer, such as direct equity, from defining or running the evaluation or approving its findings.
- Source: AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025)
- Conflict-of-interest disclosure
- A conflict-of-interest disclosure is an evaluator's published statement of relationships that could bias an evaluation, such as payment from the developer or its competitors, so readers can weigh the findings accordingly.
- Source: AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025)
Source of record
Where this definition comes from
Anthropic Responsible Scaling Policy v3.4, §3.6.1, Selecting external reviewers
“Do not have conflicts of interest with respect to Anthropic. At a minimum, a reviewing organization itself may not have a financial interest in Anthropic; and the individuals involved in conducting the review, as well as anyone above them in the reporting chain within their organization, may not have a financial interest in Anthropic or close personal relationships with anyone at Anthropic (i.e., family relationships, romantic relationships, or shared living arrangements).”
https://cdn.sanity.io/files/4zrzovbb/website/0bacdc8440ea96e62a8766d99ebe1d4eea6d5f3a.pdfMETR: OpenAI-Hugging Face incident investigation, Introduction
“Per our standard policy, we did not take payment from OpenAI for this independent assessment.”
https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/FRONTIER Act, 119th Congress (H.R. 9925, introduced, not enacted), Sec. 4(c)
“A large frontier developer shall not retain a third party if either the developer or the third party has a financial interest in the other. A large frontier developer may compensate a third party for its services but shall not condition any payment, or the amount of any payment, on the results of the third party's audit.”
https://www.govinfo.gov/content/pkg/BILLS-119hr9925ih/html/BILLS-119hr9925ih.htmAEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025), Condition 2.1
“Contingent compensation Requirement: The evaluator did not receive compensation contingent on the results of the evaluation.”
https://aievaluatorforum.org/AEF_1_Minimum_Operating_Conditions_for_Independent_Third_Party_AI_Evaluations.pdf
Crosswalk
How named organisations use this concept
| Organisation | Their term, as published | Match & verification | Source |
|---|---|---|---|
| AnthropicShadow mapping Anthropic Responsible Scaling Policy v3.4 / Advanced AI Framework / We Must Pace the Frontier (essay by Anthropic's CEO) | “RSP v3.4 §3.6.1, three selection tests: reviewers "Have significant experience and expertise regarding evaluations for dangerous AI capabilities and propensities", including knowing "ways such evaluations might be misleading (for example, alignment faking)"; "Have reputational and other incentives making them likely to be candid" ("external review parties should not be teams whose revenue, reputation and success depend entirely on Anthropic and similar companies continuing to work with them"); and "Do not have conflicts of interest with respect to Anthropic. At a minimum, a reviewing organization itself may not have a financial interest in Anthropic; and the individuals involved in conducting the review, as well as anyone above them in the reporting chain within their organization, may not have a financial interest in Anthropic or close personal relationships with anyone at Anthropic." "In selecting external reviewers, we will consult with the Board and obtain the approval of the LTBT." AAF: "independent in the sense of not having a financial interest in the developer and being free of major conflicts of interest with respect to the individuals involved in conducting the review" (p.7); avoid "evaluator shopping." Essay: embedded evaluators "can simply provide a second opinion free of commercial incentives."” Also cites the Advanced AI Framework and 'We Must Pace the Frontier'. | exactEQ confidence: high | Anthropic Responsible Scaling Policy v3.4 |
| EUShadow mapping EU GPAI Code of Practice, Safety and Security Chapter | “Appendix 3.5 "adequate qualification" test: "(1) having significant domain expertise for the systemic risk and being technically skilled and experienced in conducting model evaluations; (2) having appropriate internal and external information security protocols in place; and (3) having agreed to protect commercially confidential information, if they need access to such information." No explicit financial-conflict-of-interest test (unlike Anthropic's RSP §3.6.1); Model Report must give "an explanation of the choice of evaluator based on the qualification criteria" (Measure 7.4).” Notably lacks an explicit financial-COI test, unlike Anthropic's RSP. | closeCL confidence: medium | EU GPAI Code of Practice, Safety and Security Chapter |
| METRShadow mapping METR OpenAI-Hugging Face incident investigation / METR (About) | “"Per our standard policy, we did not take payment from OpenAI for this independent assessment"; "we accepted free API credits to conduct our experiments (also per our standard policy)", estimated at "roughly ~$400K in API credits". About page: "METR has not accepted funding from AI companies, though we make use of significant free tokens"; "We are committed to mitigating conflicts of interest where possible, and disclosing any unmitigated conflicts in our reports for these assessments."” Also cites METR's About page (https://metr.org/about). Engagement-level disclosure: no payment, but in-kind API credits declared. | exactEQ confidence: high | METR: OpenAI-Hugging Face incident investigation |
| Frontier Model ForumShadow mapping FMF Third-Party Assessments technical report | “"Methodological independence enables assessors to approach evaluations with a fresh perspective" (5, robustness); open-ended assessment "can also involve internal assessors who are sufficiently independent to internal teams running capability assessment and mitigations" (3.1); effective assessments share "independence of judgment" (1.3).” The report gives no financial, contractual or organisational test for independence. | closeCL confidence: medium | FMF Third-Party Assessments |
| AI Evaluator Forum (AEF-1)Shadow mapping AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations | “AEF-1 principle 2, "Minimized Conflicts of Interest", with conditions "Contingent compensation", "Organizational control", "CoI policy", "CoI disclosure", "Recusals" and "Separate agreements".” Verified against the AEF-1 PDF (v1, 4 Dec 2025); see the other AEF-1 row for the verbatim condition texts. | closeCL confidence: high | AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025) |
| US Congress (H.R. 9925, FRONTIER Act, not enacted)Shadow mapping FRONTIER Act, 119th Congress | “Compliance audit: "A large frontier developer shall not retain a third party if either the developer or the third party has a financial interest in the other. A large frontier developer may compensate a third party for its services but shall not condition any payment, or the amount of any payment, on the results of the third party's audit"; the audit report lists "The third party's procedures for managing conflicts of interest and any conflicts of interest of personnel involved in the audit" (Sec. 4(c)). IVO licensing rules must include "Conflict-of-interest and funding-transparency requirements, including reporting requirements regarding the IVOs' funding sources and revenue generation and self-audit requirements regarding the IVOs' personnel and leadership to ensure adequate independence from the artificial intelligence industry" (Sec. 3(c)(1)). Each IVO report carries a certification that "the IVO is free from conflicts of interest" (Sec. 5(f)).” Not enacted. | exactEQ confidence: high | FRONTIER Act, 119th Congress (H.R. 9925, introduced, not enacted) |
| US Congress (S. 5061, 119th Congress, not enacted)Shadow mapping S. 5061, 119th Congress | “The proposed AI Risk Board's conflict-of-interest policy: members have "a fiduciary responsibility to the Board, a duty to report conflicts of interest, including the appearance of a conflict of interest, and do not participate in deliberations or votes from which they personally or their employer would directly and materially benefit" (Sec. 3(b)(5)(D)(i)); the policy must "require each member to publicly disclose all relevant financial and employment relationships and include recusal procedures in the event of a conflict" (Sec. 3(b)(5)(D)(ii)); "Any member who fails to comply with the conflict of interest policy adopted pursuant to paragraph (5)(D) shall be removed from the Board" (Sec. 3(b)(3)(C)); a member denied a security clearance, or restricted from classified information, "shall be removed from the Board" (Sec. 3(b)(4)(A)(iv)).” An institutional (board-member) COI regime rather than a reviewer-selection COI regime; in CASRAI's reading, a fuller independence regime than Anthropic's RSP §3.6.1 on institutional grounds. Not enacted. | closeCL confidence: medium | S. 5061, 119th Congress |
| AI Evaluator Forum (AEF-1)Shadow mapping AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations | “Principle 2 "Minimized Conflicts of Interest": "2.1 Contingent compensation Requirement: The evaluator did not receive compensation contingent on the results of the evaluation"; "2.2 Organizational control Requirement: The system provider did not exercise organizational or financial control over the evaluator"; "2.4 CoI disclosure Requirement: The evaluator clearly disclosed potential conflicts of interest relevant to the evaluation"; "2.5 Recusals Requirement: The evaluator recused any individuals with a significant financial interest in the system provider from carrying out the evaluation."” Verified against the AEF-1 PDF (v1, 4 Dec 2025). | exactEQ confidence: high | AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations (v1, 4 Dec 2025) |
What do these codes mean?
- exact
- The source term is equivalent to this element
- close
- The source term is close but not equivalent to this element
- broad
- The source term is broader than this element
- narrow
- The source term is narrower than this element
- none
- No mapping claim — used for false-friend and declared-but-undefined rows
- EQ
- Equivalent
- CL
- Close
- BR
- Source is broader than the element
- NR
- Source is narrower than the element
- FF
- False friend — same or similar label, different meaning
- DU
- Declared but undefined by the source
- UV
- Unverified
Related, not mapped
Pointers that are not crosswalk claims
These sources mention this concept but do not define or map it clearly enough to count as a crosswalk row — noted here so the research is visible without overstating it as a mapping.
- OpenAI
Proposal that CAISI certify third-party assessors: "independent assessment certification" (blueprint p.5) -- a pointer, not a mapping; this is an OpenAI proposal, not current practice at CAISSI (formerly CAISI).
A Blueprint for a Federal Framework - Google DeepMind
HuBREC: "a research ethics committee run within Google DeepMind but staffed and chaired by academics from outside the company" (p.21) -- a pointer, not a mapping.
Gemini 3.7 Flash FSF report - Meta
Evaluations "held out from teams conducting training to reduce the risk of overfitting" (§4.1) -- internal segregation; a pointer, not a mapping.
Meta Advanced AI Scaling Framework v2 - US Government
Certification of third-party assessors by CAISSI (formerly CAISI) is an OpenAI proposal (see the OpenAI pointer above), not current practice -- a pointer, not a mapping.
A Blueprint for a Federal Framework
Gap
California SB 53 and New York's RAISE Act (as amended by S8828) ask developers to describe third-party involvement but contain no independence or conflict-of-interest test for evaluators. The EU Code of Practice tests qualification and confidentiality rather than financial ties. No source defines a common disclosure format, so independence is currently stated engagement by engagement.







