Written and maintained by CASRAI Editorial Board
Last updated
Compliance offices are being sold AI on the promise that it will read the regulations so nobody else has to. That promise is backwards: a general-purpose chatbot is worst at exactly the thing a compliance office needs most — stating what a rule currently requires, and being wrong in a way somebody can catch. But there are four jobs where AI genuinely earns its place, and one architecture that makes it safe enough to use for them.
The four jobs worth automating
Almost every credible AI use case in a research compliance office falls into one of four buckets, and they are not equally suited to it.
| Job | What AI is actually doing | How well it holds up |
|---|---|---|
| Regulatory change monitoring | Ingesting new agency publications, filtering to those touching your institution, summarising what changed | Strong. Filtering and summarisation over documents you supply. Nothing has to be recalled from memory. |
| Answering internal policy questions | Retrieving the relevant passage from your policy corpus and stating what it says, with a link | Strong, if built on retrieval. Dangerous on a raw model with no corpus behind it. |
| Triaging disclosures | Routing, deduplicating, flagging incomplete submissions, surfacing prior-year inconsistencies for review | Useful as a sorter, never as a decider. The determination is a human judgement with institutional liability attached. |
| Drafting SOPs | Producing a structured first draft from an outline, a template, and the governing regulation | Useful for structure, not substance. Every citation in the draft must be verified before it nears a controlled document system. |
Notice the pattern. AI performs well where the authoritative text is placed in front of it and the task is to find, filter or restructure. It performs badly wherever the task requires the model to know something — a threshold, a deadline, a current requirement — from its own parameters.
The failure modes that matter in a compliance context
Every generative model has these. In most settings they are an annoyance. In a compliance office they are the difference between a defensible file and a finding.
Hallucinated regulation
A language model generates text that is statistically plausible. A plausible-sounding regulatory citation — a CFR section that does not exist, a real section that says something different, a form number in the right format for the right agency but wrong — is exactly what these systems produce fluently and confidently. That is more dangerous than a hallucinated fact in ordinary prose, because a citation carries an implicit invitation not to check it: it looks like verification has already happened.
The compliance-specific hazard is that the wrong answer is usually almost right. A model rarely invents a fictional agency. It gives you a real regulation with a subtly wrong threshold, or the superseded version of a requirement, phrased with the same confidence as the correct one.
Stale training data
Every model has a training cutoff, and regulatory text moves after it. The scale is easy to underestimate. In calendar year 2025 alone, the Federal Register published 2,441 final rules and 1,498 proposed rules across all federal agencies — figures retrieved directly from the Federal Register public API. Only a small fraction touches research administration, but a small fraction of nearly four thousand documents is still more than any office reads unaided, and a model trained before a change cannot signal that it is answering from a superseded state of the world.
Worse, staleness is silent. A model does not say “this may be out of date.” It answers in the present tense.
Inability to cite
A base model cannot tell you where an answer came from, because there is no “where” — it was generated, not retrieved. Asking for a citation after the fact produces a generated citation, which is a worse thing than a real one. A compliance office’s output is not the answer; it is the answer plus the trail. An uncited assertion cannot be entered into a file, defended at an audit, or handed to a PI who will reasonably ask “says who?”
No refusal behaviour
The most consequential gap. Asked something outside its knowledge, a general model answers anyway, because answering is what it optimises for. A compliance tool needs the opposite default: when the governing text is unavailable, it must say so and stop. A tool that never says “I don’t have that” gives you no way to separate its reliable answers from its unreliable ones — which makes all of them unreliable.
Why retrieval with citations is the architecture that answers this
Retrieval-augmented generation inverts the order of operations. Instead of asking a model what it knows, the system first searches a defined corpus, retrieves the passages matching the question, and asks the model to answer using only those passages. The answer is constructed from text that was fetched, not recalled. Each failure mode above maps onto a property this provides:
| Failure mode | What retrieval changes |
|---|---|
| Hallucinated regulation | The model answers from a passage in front of it, and that passage is shown to the reader. A fabrication is visible as a mismatch between claim and cited source. |
| Stale training data | Currency becomes a property of the corpus, not the model. Re-ingesting sources on a schedule updates what the system can say without retraining anything. |
| Inability to cite | The citation is the retrieved document — a real reference by construction, not a generated one. |
| No refusal | When retrieval returns nothing relevant, the system has a defined state to report — “not in the corpus” — rather than falling back on the model’s parameters. |
What retrieval does not fix. Be precise about the residual risk, because vendors will not be. Retrieval guarantees a cited passage exists and was used; it does not guarantee the passage is the right one, that it is current, or that the model read it correctly. Three failure modes survive: retrieving a superseded document still sitting in the corpus; a corpus gap reported as an absence of a requirement rather than an absence of coverage; and a correct retrieval paired with a summary that subtly misstates it. Citation makes all three checkable — that is the claim worth making, and it is not the same as making them impossible.
What to require before adopting any AI tool
Treat this as procurement criteria, not aspirations. Ask for each in writing, and treat a vendor’s inability to answer plainly as the answer. This complements — and does not replace — a broader institutional governance framework; the NIST AI Risk Management Framework (NIST AI 100-1, released 26 January 2023, voluntary) organises institutional AI risk around four functions — Govern, Map, Measure and Manage — and we cover its institutional application in the NIST AI RMF for research institutions.
1. Source citation at the claim level
Not “sources consulted” appended to the bottom of a response, but a specific citation attached to a specific claim, resolving to a document you can open. Test it adversarially: ask questions whose answers you already know, and check that the cited passage actually supports the sentence it is attached to. A tool that cites a plausible document not containing the claim is worse than one that does not cite at all, because it manufactures unearned confidence.
2. Auditability
You need to reconstruct, months later, what the tool said and on what basis: a query log, the retrieved sources per answer, the response, a timestamp, and the user. This is the instinct behind audit trail review procedures and the 21 CFR Part 11 requirement at §11.10(e) for secure, computer-generated, time-stamped audit trails; the underlying principle of data integrity applies with equal force. Ask about reproducibility too: if the same question returns materially different answers on different days with no corpus change, you cannot audit it meaningfully.
3. Data handling
Compliance offices handle unusually sensitive material — draft disclosures, allegations, investigation records, unpublished protocols. Establish before any pilot: where queries are processed, whether inputs are retained and for how long, whether they train the vendor’s models, who at the vendor can read them, and what contract binds it. We cover the mechanisms in zero data retention for AI tools in research and getting a BAA for AI tools used in research. If the workflow touches identifiable health information or export-controlled material, that determination belongs to your privacy and export offices before the pilot.
4. A documented refusal behaviour
Require the vendor to state in writing what the system does when it cannot answer — then verify it by asking questions deliberately outside its coverage. You are looking for an explicit, legible refusal. A tool that produces a confident general-knowledge answer when its corpus has nothing has quietly reverted to being a chatbot, and every other guarantee in the contract is downstream of that behaviour.
Where AI does not belong in the loop
Draw this line explicitly in your acceptable-use policy rather than leaving it to individual judgement. AI should not make, or appear to make, a determination — the institutional decisions that carry regulatory weight and personal accountability. A model may surface that a disclosure looks inconsistent with a prior filing; a human decides whether that is a conflict requiring a management plan. A model may summarise a rule; a human decides whether the institution is compliant with it. A model may draft an SOP section; a qualified owner approves it into the controlled document system under their own name.
The practical test: if the output would be signed, filed with an agency, relied on by a PI, or produced in an audit, a named human must own it and must have checked the cited source. The failure mode is rarely a bad tool — it is a good tool used one step past where anyone verified it. Our framework for choosing and governing LLMs for research covers the parallel researcher-facing questions of confidentiality, disclosure and reproducibility.
An adoption sequence that works
- Pick one bounded job. Regulatory change monitoring is the usual starting point: the output is a summary a human reads before acting, so an error is caught cheaply.
- Build the evaluation set first. Twenty to fifty real questions your office has answered, with correct answers and sources recorded. Write these before you see any demo, or the demo will shape them.
- Run the adversarial half. Questions with no correct answer, questions about superseded requirements, questions just outside coverage. What the tool does with these tells you more than what it handles well.
- Check citations by hand. Open every cited source and confirm it says what the answer claims — not a sample, all of them, at pilot scale.
- Write the policy before rollout. Permitted uses, prohibited uses, the determination boundary, what must be disclosed to researchers, and who owns verification.
An example of the cited-retrieval pattern
Disclosure: Ask CASRAI is our own product, offered here as an illustration of the architecture rather than a neutral recommendation. It is a retrieval-augmented question-answering system specialised for research administration, running over CASRAI’s published corpus plus six external feeds — the U.S. Federal Register, a research and grants slice of it, Grants.gov, Regulations.gov, NSF News and UKRI — re-ingested by a scheduled daily job, which is how currency is maintained without retraining anything. It cites a source for each factual claim and answers “I don’t have that in the CASRAI corpus” rather than guessing. It is a paid product, it answers questions rather than drafting documents, and it does not cover every funder or regulator. Hold any tool you evaluate, this one included, to the four requirements above. For how general assistants handle adjacent tasks, see chat with PDF tools for researchers and Claude for research.
Frequently asked questions
Can AI replace a research compliance officer?
No, and the reason is structural rather than a matter of current capability. Compliance work terminates in determinations that carry institutional and personal accountability — a conflict management plan, a certification to an agency, a signature on a controlled document. Accountability cannot be delegated to a system that cannot be held responsible. AI changes how the preparatory work gets done, not who owns the decision.
Is it safe to ask a general chatbot about a federal regulation?
Only as a way to find the actual text, never as the answer itself. The risk is not that it will be obviously wrong — it is that it will be plausibly and confidently wrong about a threshold, a deadline or a version. Treat every output as an unverified lead and go to the primary source before acting on it.
What is the difference between a retrieval-based tool and a general AI assistant?
A general assistant answers from patterns learned during training. A retrieval-based tool searches a defined corpus first and answers from what it found, showing the source. So the retrieval tool can be updated by updating its corpus, its citations point to real documents, and it has a coherent way to say it does not know. A general assistant has none of those three properties by default.
How do we keep an AI tool current with regulatory change?
Currency should be a property of the corpus and its ingestion schedule, not of the model. Ask any vendor two questions: which sources are ingested, and how often. If the answer is that the model “knows” current regulation, that is a training-cutoff answer wearing different clothes, and it will silently degrade.
Do we need to disclose AI use in compliance work?
Check the specific obligation rather than assuming a general one. Disclosure requirements attach to defined contexts — journal and funder policies on AI-assisted manuscript preparation, some agency submission rules, and increasingly institutional policy. Frameworks are still moving: the EU AI Act’s implementation timeline was amended during 2026, so check the deferred dates against the Commission’s own timeline before anything filing-critical relies on them.
What should we pilot first?
Regulatory change monitoring. The source documents are public, so the data-handling review is straightforward; the output is a summary a human reads before acting, so errors surface cheaply. Policy question-answering is a reasonable second step once you have a verified corpus. Disclosure triage should come last, and only as a sorting aid.
