Written and maintained by CASRAI Editorial Board
Last updated
What explainable AI (XAI) means
Explainable AI (XAI) is the property of an AI system, and the set of techniques used to achieve it, that lets a human understand why the system produced a particular output — which inputs mattered, how they were weighted, and what would have changed the result. IBM’s working definition frames it plainly: XAI is “a set of processes and methods that allows human users to comprehend and trust the results and output created by machine learning algorithms.” That comprehension is the whole point. A model that is merely accurate but cannot be interrogated is a black box; a model with XAI applied to it can be asked “why this answer, and not another one,” with an answer a human can actually evaluate.
XAI is a property and a toolkit, not a single technique. It spans everything from choosing an inherently interpretable model (a shallow decision tree over a deep network, when accuracy allows it) to applying post-hoc explanation methods to a model that is not interpretable by design. Most of the current regulatory and enterprise attention — and most of what this guide covers — is about the second case: explaining a model that is already built, already deployed, and too complex to read directly.
How XAI methods work, at a conceptual level: SHAP, LIME, and model cards
Three tools come up constantly once a team moves from “we want explainability” to “which method do we actually run.” They solve different problems and are not interchangeable.
SHAP (SHapley Additive exPlanations) is a game-theoretic approach to explaining a single prediction. It draws on Shapley values — a method from cooperative game theory for splitting credit fairly among players — and applies that idea to a model’s input features, computing how much each feature pushed a specific prediction up or down relative to a baseline. SHAP is model-agnostic (it treats the underlying model as a black box and does not need access to its internals) and produces local explanations: an accounting of one prediction, not a description of the model’s behavior as a whole.
LIME (Local Interpretable Model-Agnostic Explanations) tackles the same local-explanation problem from a different angle: it perturbs the input around the specific case being explained, observes how the model’s output changes, and fits a simple, interpretable model (typically linear) to that local neighborhood. The simple model isn’t a stand-in for the whole system — it’s only trustworthy in the immediate vicinity of the prediction being explained — but that’s enough to answer “which features drove this particular decision.”
Model cards answer a related but different question: not “why did the model say this,” but “what is this model, and where is it safe to use.” A model card is a short, standardized document published alongside a model reporting what it is, how it was trained, its measured performance (including how that performance breaks down across demographic and contextual slices), known limitations, and intended versus out-of-scope uses. CASRAI’s dictionary carries the full definition at its model card entry rather than repeating it here — the short version for this guide is that a model card is documentation-level explainability (what the model is and where it was validated), while SHAP and LIME are prediction-level explainability (why it said this, this time).
Explainable AI vs AI transparency: how they differ
The two terms get used interchangeably in casual writing, and the distinction matters once a compliance team has to map obligations to specific documents. AI transparency is the broader disclosure discipline: what a provider tells deployers, regulators, and the public about a system — its intended use, training data sources, performance characteristics, known limitations, and how to interpret its logs. Explainability is one specific capability that sits underneath the transparency umbrella: the ability to account for an individual output, not just describe the system in general terms. A provider can be highly transparent (a detailed, honest instructions-for-use document) about a model that is still hard to explain output-by-output, and a model can support strong local explanations (via SHAP or LIME) while its provider discloses very little else about it. The two are complementary, not synonyms, and the regulatory language below treats them that way: disclosure obligations sit alongside a narrower requirement that outputs be interpretable.
For the regulatory transparency regime this sits inside, see CASRAI’s explainer on California SB 53, the “Transparency in Frontier Artificial Intelligence Act,” alongside the management-system standards organizations certify against for the same underlying discipline — the NIST AI Risk Management Framework and ISO/IEC 42001. For who is on the hook for an explainability-relevant decision once a framework requires one, see CASRAI’s guide to accountable decision-makers under SB 53.
Why regulators and boards now require it
The clearest binding requirement comes from the EU AI Act. Article 13, “Transparency and provision of information to deployers,” requires that high-risk AI systems “be designed and developed in such a way as to ensure that their operation is sufficiently transparent to enable deployers to interpret a system’s output and use it appropriately.” That is an explainability requirement stated in transparency language: providers must supply instructions covering the system’s characteristics, capabilities, and limitations, the accuracy metrics and known failure conditions, and — specifically — information to enable a deployer to interpret the system’s output, not merely to know that it exists.
Inside organizations, the same pressure shows up one level down from statute, in risk governance rather than case law: enterprise AI-governance frameworks that boards and risk committees ask management to certify against, such as ISO/IEC 42001 (the AI management system standard) and the NIST AI Risk Management Framework, put explanation and interpretation of model behavior in scope as a named control area alongside data governance and incident response, rather than leaving it as a data-science-team concern with no board-level visibility. The practical effect is the same in both cases: “the model works” is no longer a sufficient answer on its own. A team now has to be able to say why it works, for this input, in terms a deployer, auditor, or board member who did not build the model can actually evaluate.
Where CASRAI’s NIKOLAI dictionary fits
CASRAI’s own NIKOLAI dictionary — an independent, unendorsed reference tracking frontier-AI-safety terminology across labs and regulators, not a standard and not something any lab or regulator has approved — does not yet catalog “explainability” as its own element among its 64 elements. Rather than force a link to a definition that doesn’t exist, it’s worth naming the closest formalization NIKOLAI does carry: its Claims & Argument track defines a Safety Case as a structured, reviewable argument built from one or more Claim records, each an atomic assertion carrying its own supporting evidence and an explicit rule for how it combines with sibling claims. That is explainability applied to a safety decision rather than to a single model output: not “trust us, it’s safe,” but a decomposable argument a reviewer can actually inspect claim by claim — the same underlying discipline SHAP and LIME apply at the level of one prediction.
NIKOLAI’s own Mapping Declarations system is a working example of that discipline applied to NIKOLAI itself. Every crosswalk row NIKOLAI publishes starts as a “shadow mapping” — CASRAI’s own unendorsed reading of a lab’s or regulator’s published framework, not an endorsement by anyone else. An organization can file a declaration to confirm, dispute, or add a mapping, but that declaration does not go live on its own say-so: every declaration, from a lab, a regulator, or an evaluator, is reviewed by a CASRAI editor before publication, and evaluator accounts additionally require manual acceptance after email verification before they can submit at all. The result is an auditable trail with each step attributable: what was claimed, by whom, and who reviewed it before it counted as more than a shadow mapping — the same properties (attribution, review, a record that survives the decision) that explainability requirements are trying to get out of a model’s own outputs.
Frequently asked questions
What is explainable AI?
Explainable AI (XAI) is the property of an AI system — and the techniques used to achieve it — that let a human understand why the system produced a specific output: which inputs mattered, how they were weighted, and what would change the result. It ranges from choosing an inherently interpretable model to applying post-hoc methods like SHAP or LIME to a model that was not built to be read directly.
What does “explainable AI (XAI)” mean as an acronym?
XAI is simply the standard abbreviation for explainable AI. The two terms are used interchangeably in research and regulatory writing; “XAI” tends to show up in technical and academic contexts, and “explainable AI” in policy and governance writing, but they refer to the same property.
Is explainable AI the same as AI transparency?
No. AI transparency is the broader disclosure discipline — what a provider tells deployers and regulators about a system’s design, training data, and performance. Explainability is a narrower capability underneath that umbrella: the ability to account for a specific output, not just describe the system in general. A system can be transparently documented and still hard to explain output-by-output, and vice versa.
What are SHAP and LIME, and how do they differ?
Both are model-agnostic methods for explaining a single prediction (local explanations), not the model’s overall behavior. SHAP uses Shapley values from game theory to allocate credit for a prediction across input features. LIME perturbs the input around the case being explained and fits a simple, locally interpretable model to approximate the original model’s behavior in that neighborhood. Neither explains the model globally; both explain one decision at a time.
Do model cards make a model explainable?
Not on their own. A model card documents what a model is, how it was trained, and where its measured performance holds up or breaks down — documentation-level transparency. It does not explain any individual prediction the way SHAP or LIME does. The two are complementary: a model card tells you whether the model is appropriate to use here; a local-explanation method tells you why it produced this particular result.
Why are regulators requiring explainability now?
The EU AI Act’s Article 13 requires high-risk AI systems to be built so that their operation is “sufficiently transparent to enable deployers to interpret a system’s output and use it appropriately” — an explicit interpretability requirement, not just a documentation one. Enterprise AI-governance frameworks such as ISO/IEC 42001 and the NIST AI Risk Management Framework carry the same expectation into board-level risk oversight, treating explanation of model behavior as a named control area rather than a data-science-only concern.
