Building a Registry That Can’t Accidentally Lie

Most open registries that let outside organizations speak for themselves eventually hit the same design problem: once you let a lab, a regulator, or an evaluator attach a claim to their own entry, you’ve created a channel that can lie — not maliciously, necessarily, but casually, incompletely, or under review pressure nobody applied. A registry that never lets anyone speak for themselves stays honest by staying useless. A registry that lets anyone speak for themselves stays useful by becoming a liability. The interesting design work is in between.

This is a case study of how CASRAI’s own NIKOLAI project handled that problem for its Mapping Declarations mechanism — the process by which a lab, regulator, or evaluator can confirm, dispute, or add to how NIKOLAI has crosswalked their published materials. It isn’t a how-to for filing a declaration; that document already exists at Mapping Declarations: How Organizations Verify and Confirm Their Own AI Safety Terminology in NIKOLAI. This is the design story underneath it — what NIKOLAI built, what it deliberately didn’t, and where it says so on its own pages instead of pretending otherwise. If you’re building a registry, a dictionary, or any structured dataset that will eventually carry someone else’s organizational attribution claim, the pattern here is meant to transfer.

The problem: a claim needs a route from “unverified” to “trusted,” and each step has to answer a different question

Every row in a NIKOLAI crosswalk starts life as what NIKOLAI calls a shadow mapping: CASRAI’s own reading of how a term is used in an organization’s published materials, built without that organization being consulted. Shadow mappings are useful — they’re how the dictionary gets built at all — but they’re also, structurally, guesses. The Mapping Declarations mechanism (NIKOLAI’s Track N10, Assurance Roles) exists to let an organization move a specific row from “CASRAI’s guess” to “this organization’s own confirmed statement,” and to let a third party formally contest a mapping it thinks is wrong.

The design problem is that “let an organization speak for itself” is actually three separate questions stacked on top of each other, and NIKOLAI’s team found early that collapsing them into one gate is exactly how a registry ends up publishing a false attribution:

1. Is this account controlled by someone at the organization? A real but weak signal — an intern’s inbox clears it as easily as a general counsel’s.
2. Should this account type be trusted to act in this specific role at all? A different question with a different answer depending on what the account is claiming to be.
3. Is this specific claim, once made, actually true and correctly recorded? The question none of the above answer, and the only one that determines whether anything gets published.

NIKOLAI answers each with a separate, independent gate, and none of them alone is sufficient. That separation — not any single control — is the actual mechanism worth borrowing.

Gate one: domain-matched signup, fast for everyone, proof of nothing but control of an inbox

An organization signs up for a NIKOLAI account with an email address at its own domain. This is automatic and instant for every account type — a lab, a regulator, or a third-party evaluator all clear it the same way. It’s a real signal, not zero verification: it rules out an anonymous or unaffiliated party filing on a company’s behalf. But it answers only the first question above, and the system is explicit internally that domain control is not proof of authority to speak for an organization, let alone proof that any specific claim is accurate.

Gate two: a manual admin gate, but only for the role that needs a higher bar

This is the design decision worth naming on its own. Domain verification alone is enough to make a lab or regulator account active, because the claim a lab account will go on to make is about its own framework — the organization is the world’s authority on how it uses its own terms. An evaluator account is different: NIKOLAI’s own account status stays at pending_admin_approval for the evaluator role even after domain verification clears, requiring a CASRAI editor to accept it before the account can act. The reasoning, stated directly in NIKOLAI’s own build notes: evaluators are the parties NIKOLAI’s assurance track exists to hold to a higher bar than the labs they might one day corroborate or dispute, so an evaluator claiming standing to judge someone else’s declaration doesn’t get to self-certify that standing the way a lab describing its own framework does.

The general version of this rule: the same verification mechanism does not have to clear every account type to the same trust level. Ask what each account type’s eventual claim actually asserts, and gate the ones that assert something about a third party more tightly than the ones that assert something about themselves.

Gate three: nothing publishes without a human reviewer, regardless of account type

Neither gate above is what stops a false attribution from actually reaching a live page. That’s a third, separate control: every filed declaration — from a lab, a regulator, or an approved evaluator alike — sits at a pending-review status until a CASRAI editor with manage_options capability approves or rejects it through a dedicated REST endpoint. This is, concretely, the only code path anywhere in the system that can ever set a crosswalk row’s declared_by field to a non-null value; NIKOLAI’s own element importer is separately, unconditionally blocked from ever doing so, on purpose, so the seed pipeline that built the dictionary in the first place can never masquerade as an organization’s own declaration. The approval itself is recorded durably — who reviewed it and when — rather than left as an implicit side effect of a status change.

That’s the shape of the answer to “how do you let outside organizations speak for themselves without letting them lie”: you don’t skip straight from “verified account” to “published claim.” You put a mandatory, logged human decision between the two, for every account type, with no fast path that bypasses it — not even for the organizations the registry trusts most.

The honesty gradient: what happens when you name six roles before you’ve built all six

NIKOLAI’s Track N10 names six roles in this process — CASRAI Declaration Reviewer, Declaring Representative, Declaration Disputant, Corroborating Evaluator, Declaration Accuracy Steward, and Declaration Role Statement — and states on each element’s own page exactly how real that role currently is in the running system. That gradient, and NIKOLAI choosing to publish it rather than smooth it into six equally-confident-sounding definitions, is the part of this design most worth studying.

Fully wired: CASRAI Declaration Reviewer

The reviewer role is the one with no gap between what’s described and what’s built. It’s gated to a real editor account (current_user_can('manage_options')), it’s the sole code path that can flip a row from shadow to declared, and every review action durably records who reviewed it and when. NIKOLAI’s own page for this element notes that the role has no CRediT counterpart at all — CRediT’s contributor-role taxonomy deliberately excludes peer review from its scope, so this is a role N10 had to invent rather than adapt.

Implicitly real: Declaring Representative

Every declaration submission records the WP user ID that filed it — in practice, that user is this role’s incumbent for that one declaration. But NIKOLAI is explicit that this shouldn’t be read as a fully realized, named-person role in the sense CRediT roles are: there’s no persistent, labeled “Declaring Representative” record, and the public-facing output shows only the organization’s name, not the individual who filed on its behalf.

Fixed the same day it broke: Declaration Disputant

This is the most instructive story on the page, and NIKOLAI states it plainly rather than papering over it. At initial release, the approval logic ran an identical status flip for a “confirm” and a “dispute” — a dispute published looking exactly as authoritative as an endorsement, with the disputed content left untouched and nothing on the page indicating anything had been contested. That’s a worse outcome than not having a dispute mechanism at all: it let a real objection get recorded in a form indistinguishable from agreement. Caught and fixed the same day: an approved dispute now replaces the targeted row’s content with the disputing organization’s own corrected statement, sets a permanent disputed: true flag distinct from a plain confirmation, and renders with its own styling on the live element page — a red “Disputed & corrected” badge, distinct from the green “Declared” badge a confirmation gets and the amber “Shadow mapping” badge an unconfirmed row still carries.

No code path at all: Corroborating Evaluator

An evaluator independently corroborating another organization’s declaration is described as a role in N10’s own definition — but NIKOLAI states directly that it isn’t merely unused, there is currently no code path that could record one even if an evaluator tried. It isn’t wired into the submission form at all.

The most purely aspirational: Declaration Accuracy Steward

The role responsible, once a declaration is live, for keeping it accurate over time. NIKOLAI’s own language is unusually direct about how thin this one is: there is no stored “accuracy steward” field of any kind, not even an implicit one the way Declaring Representative has one — no periodic-review mechanism and no owner-of-record exists in the data model today. NIKOLAI calls this out as a materially weaker claim than “no incumbent yet,” and says so on the page rather than leaving a visitor to assume the gap is smaller than it is.

A record type, not a role: Declaration Role Statement

The sixth element isn’t a person or account type at all — it’s the record that would capture which named person or organization held which of the other five roles against one specific declaration, and when. As of this writing none has ever been recorded, for the plainest possible reason: no Mapping Declaration has yet been filed against the live system. It’s a proposed record type, stated as exactly that, not a feature quietly implied to already exist.

Why model this on CRediT at all, given NIKOLAI deliberately doesn’t reuse CRediT’s roles

Track N10 cites ANSI/NISO Z39.104-2022 — the Contributor Roles Taxonomy, known as CRediT — as its structural model, but it’s worth being precise about what that means, because NIKOLAI’s own build notes are explicit that this is not a literal reuse. CRediT defines fourteen standardized roles describing how an individual contributed to a piece of scholarly work — conceptualization, data curation, writing, and so on — and N10 crosswalks none of it, because a research-authorship taxonomy has no roles for “who may file a Mapping Declaration” or “who may dispute one.” What N10 borrows is the pattern CRediT established — a fixed, named vocabulary of roles instead of a single undifferentiated “contributor” or “party” label — applied to a governance question CRediT was never built to answer.

That gap is also the clearest external justification for building a role like Declaration Disputant at all: CRediT itself has no dissent or dispute mechanism anywhere in its fourteen roles. There’s no CRediT role for “formally contests another contributor’s credited role,” so N10’s Declaration Disputant isn’t filling a gap CASRAI invented to have something to build — it’s naming a function the model it’s borrowing from doesn’t address, for the good reason that disputing a co-author’s contribution and disputing a lab’s own safety-framework terminology are different problems with different stakes.

One correction worth being precise about, since it bears directly on how much weight this comparison can carry: CASRAI originated CRediT and stewarded it from 2014, but in 2020 formally transferred that stewardship. CASRAI’s own CRediT overview and its federation page are both explicit that NISO — not CASRAI — is CRediT’s current steward, publishing it as ANSI/NISO Z39.104-2022. N10 citing CRediT as a structural model is citing an external standard CASRAI helped build and no longer runs, not reusing something still under CASRAI’s own roof. That distinction matters for exactly the audience this piece is written for: if you’re modeling a governance mechanism on an existing taxonomy, say precisely who owns the thing you’re borrowing from.

The decisions worth taking if you’re building this for your own registry

Stripped of NIKOLAI specifics, three decisions in this build are the ones worth deliberately making, not defaulting into, in any registry that will eventually carry someone else’s organizational attribution claim:

• Don’t gate every account type identically. Work out what each account type’s eventual claim actually asserts — about itself, or about a third party — before deciding how hard it should be to activate. An account that will only ever describe its own organization and an account that will sit in judgment of someone else’s declaration are not the same trust problem, even if both clear the same email-domain check.
• Put a mandatory, logged human gate between “verified” and “published,” with no exceptions for trusted account types. The domain check and the evaluator approval both answer “can this account act at all.” Neither answers “is this specific claim true and correctly recorded.” That’s a separate question, and NIKOLAI’s design puts a real person’s judgment — not a stronger automated check — between a claim and its publication, for every submitter, every time.
• When a role in your design isn’t built yet, say so on the page, at the level of precision the gap actually deserves. NIKOLAI doesn’t collapse “no incumbent yet” (Declaring Representative), “no code path at all” (Corroborating Evaluator), and “not even a data-model field exists” (Declaration Accuracy Steward) into one vague “coming soon.” Those are three different amounts of unbuilt, and a reader deciding whether to trust the mechanism needs to know which one they’re looking at. The alternative — announcing a role as though it functions, because the taxonomy names it — is exactly the kind of registry defect this whole mechanism was built to prevent happening to someone else’s data; there’s no reason to reintroduce it into your own documentation of how the mechanism works.

NIKOLAI’s assurance layer hasn’t had its first real Mapping Declaration filed against it yet, by its own account. Two of its six named roles have no code behind them yet, and it says so. That combination — a real, working gate for the claims the system already handles, and an honest gap note for the ones it doesn’t — is a more useful thing to have built first than a complete-looking system that quietly hides which parts of it are load-bearing.