Silent Revision: Most AI Framework Changes Hidden

Last verified: September 20, 2026. A developer publishes a safety framework, then quietly revises it months later without saying what changed — and according to a new study, that is the norm, not the exception. A paper posted to arXiv on September 8, 2026, traces 710 individual commitment instances across twelve frontier AI developers’ safety frameworks and finds that roughly two-thirds of the material changes made to those commitments never show up in the developer’s own published account of what changed. CASRAI’s own NIKOLAI project — an independent, unendorsed reference for frontier-AI-safety terminology — already has an element built for exactly this gap, and this page opens with that connection because it is the whole reason the finding matters operationally, not just academically: NIKOLAI’s Framework Update Protocol element (N9) already treats “whether a changelog is required” as a distinct, trackable property, and this study is the first evidence of what happens when that property is left unset.

The Study, in One Paragraph

“Silent Revision: Measuring Undisclosed Change in the Safety Frameworks of Frontier AI Developers,” by Louis Yiven Zhu, was submitted to arXiv (2609.08789) on September 8, 2026. It builds a versioned, hash-pinned corpus of every public version of the safety frameworks published by twelve developers, together with each provider’s own changelog, redline, or announcement of what changed. Zhu traces 710 commitment instances across twelve consecutive version pairs, codes them against a frozen codebook, and individually adjudicates 244 of them. The paper’s own term for the core metric is the silent revision rate: the share of material changes to a framework’s commitments that the developer’s own published account does not identify.

What the Study Found

Headline rate: 67% of material changes are silent under a strict disclosure standard (95% CI: 62-72%), falling to 53% under a lenient standard and 49% when measured at section-level granularity.
Format matters more than length: narrative-style change announcements run 74% silent, against 63% for itemized changelogs. Account length in words barely predicts silence — the format of the account does.
Direction matters: 77% of the traced changes weaken or remove a commitment rather than strengthen one, and in seven of the eight developer pairs examined, weakenings were more often silent than strengthenings.
The corpus: 710 commitment instances traced across twelve consecutive version pairs; 244 individually adjudicated against a frozen codebook.
The policy argument: current publication duties — California’s SB 53 among them — require a developer to publish a modified framework and a justification for the modification. Zhu’s paper argues that a justification explains why a framework changed, but only an itemized enumeration of what changed makes the revision auditable, and that one provider already meets an enumeration duty voluntarily, though incompletely.

Why “Justification” Isn’t the Same as “Enumeration”

This is where the paper’s finding intersects directly with existing law rather than staying theoretical. California’s SB 53 already requires a covered developer to publish a modified framework and a justification for that modification within 30 days of a material change — but a justification is a narrative account of why something changed, not a line-by-line record of what changed. Zhu’s paper measures exactly that distinction empirically: narrative-format disclosures were silent on 74% of material changes, while itemized changelogs — a structurally different kind of account, one that lists individual line items rather than explaining a decision — were silent on 63%. Both numbers are high. But the gap between them is the paper’s central policy argument: the statutory remedy that already exists (a justification requirement) specifies the wrong artefact for the actual problem (auditability), and the fix the paper proposes is a distinct enumeration duty — a requirement to itemize each change, not just explain the framework’s current state.

Where NIKOLAI Fits In

NIKOLAI, CASRAI’s own independent, unendorsed dictionary of frontier-AI-safety elements, already has a home for this exact distinction. Framework Update Protocol, element N9 in the Commitments and Governance track, is defined as an if-then rule set governing when and how a safety framework may be revised — and among the properties it tracks are the declared review cadence, who may propose and approve a change, the publication deadline after approval, and, explicitly, whether a changelog is required. Verified directly against NIKOLAI’s live element page on September 20, 2026: the element’s own reference notes already cite the EU’s GPAI Code of Practice as the more demanding comparator, since it requires notifying the AI Office of a change within five business days, “including a changelog… along with a version number and the date of change” — structurally the itemized-changelog format Zhu’s paper measures at the lower, 63% silence rate, versus the narrative-justification format SB 53 requires, measured at 74%.

That is a genuine, substantive tie-in, not a forced one: NIKOLAI’s element already distinguishes changelog-based accounts from narrative ones as a trackable property, and Zhu’s paper is the first empirical evidence CASRAI has found that the distinction NIKOLAI flags actually predicts how much a developer’s account leaves out. As with every NIKOLAI crosswalk, this is a shadow mapping — CASRAI’s own independent reading connecting a published element definition to a published research finding — not a mapping any of the twelve developers in Zhu’s corpus, or NIKOLAI itself, has confirmed through a formal Mapping Declaration. NIKOLAI’s Framework Update Protocol element is currently tagged proposed (nikolai-v0.1), and this page will be revisited if that status changes or if a developer’s own account contradicts the reading above.

Frequently Asked Questions

What is the “silent revision rate”?

It is the metric Zhu’s paper introduces: the share of material changes to a safety framework’s commitments that the developer’s own published account — its changelog, redline, or announcement — does not identify. The paper measures it at 67% under a strict standard (95% CI: 62-72%), 53% under a lenient standard, and 49% at section-level granularity.

Are itemized changelogs more transparent than narrative announcements?

According to the study, yes, though neither is close to complete. Narrative-format change announcements were silent on 74% of material changes; itemized changelogs were silent on 63%. Word count of the account had little independent effect.

Do the undisclosed changes tend to make frameworks stronger or weaker?

Weaker. 77% of the changes Zhu’s team traced weakened or removed a commitment rather than strengthened one, and in seven of the eight developer pairs studied, weakening changes were more often silent than strengthening changes.

Does current law require an itemized changelog?

Not uniformly. California’s SB 53 requires a published justification for a material modification within 30 days, which the paper treats as a narrative-format duty. The EU’s GPAI Code of Practice goes further for signatories, requiring a changelog with a version number and date of change within five business days of a modification. Zhu’s paper argues publication duties generally should add a distinct enumeration duty rather than relying on justification alone.

Is NIKOLAI’s connection to this study an official or endorsed mapping?

No. It is a shadow mapping — CASRAI’s own independent reading connecting NIKOLAI’s Framework Update Protocol element to Zhu’s published findings — not a mapping confirmed by Zhu, by any of the twelve developers studied, or by NIKOLAI through a formal Mapping Declaration.

Silent Revision: A New Study Finds Most Safety-Framework Changes Go Undisclosed