Assessing Third-Party AI Vendor Risk

Last verified: September 20, 2026. Assessing third-party AI vendor risk means checking a specific set of things before you sign, not just running your normal SaaS security review with “AI” added to the label: where the vendor’s model’s training data came from, whether the vendor discloses a model card or system card, whether your data trains their model by default, what an AI-output indemnification clause actually requires you to do to stay covered, whether an AI sub-processor sits behind the vendor’s own sub-processors, and what audit rights you have if the vendor’s AI-specific claims turn out to be wrong. Most of this sits outside a standard vendor security questionnaire, which is why it gets skipped. CASRAI’s own NIKOLAI dictionary defines several of the terms this checklist uses, including a Deployment Surface element in its N1 track — more on that below.

Why a Generic SaaS Vendor Review Misses This

A standard SaaS vendor security review is built to answer questions a conventional software product raises: where is data hosted, who can access it, is there a SOC 2 report, what’s the breach-notification SLA, what does the master service agreement say about liability. Those questions are still necessary for an AI vendor — they just aren’t sufficient, because an AI feature introduces a second layer of risk that a conventional security review has no vocabulary for.

That second layer exists because most vendors selling “AI-powered” features aren’t training their own foundation model. They’re building a product on top of someone else’s — usually a frontier lab’s API (OpenAI, Anthropic, Google) or an open-weight model they’ve fine-tuned. That means the vendor’s own security posture and the underlying model’s behavior are two separate things to evaluate, and a vendor can pass every item on a conventional questionnaire while the model underneath is a complete unknown: what it was trained on, whether your prompts and outputs feed back into someone else’s training run, and what happens if the model produces an infringing or defamatory output your organization is exposed for.

NIST’s own AI Risk Management Framework treats this as a distinct, named category of difficulty, not an edge case: “Risks related to third-party software, hardware, and data: Third-party data or systems can accelerate research and development and facilitate technology transition. They also may complicate risk measurement. Risk can emerge both from third-party data, software or hardware itself and how it is used. Risk metrics or methodologies used by the organization developing the AI system may not align with the risk metrics or methodologies us[ed] by the organization deploying or operating the system.” That’s the whole problem in one paragraph: the vendor measured its own risk one way, and you have no visibility into how.

Training-Data Provenance: What to Actually Ask

“Where did the training data come from” is the question every AI vendor gets, and it’s usually answered with a paragraph of marketing language. Ask it as four separate, answerable questions instead:

• Whose model is it? Is this the vendor’s own trained model, a fine-tune of an open-weight model, or a wrapper around a third-party API? Get the specific model name and version, not “proprietary AI” or “a leading large language model.”
• What data trained the base model, and does the vendor know? If the vendor is reselling access to a frontier lab’s API, the honest answer is often “we don’t have visibility into that either” — which is a legitimate answer, but it means the training-data provenance question has to go to the frontier lab’s own published documentation (a model card or system card), not the vendor.
• What data trains the fine-tune or the retrieval layer, if there is one? This is the part the vendor does control, and where “customer data” answers matter most.
• Is there a data-provenance record you can inspect, not just a claim? NIST’s Generative AI Profile suggests exactly this as a concrete governance action: system inventory entries should include “Data provenance information (e.g., source, signatures, versioning, watermarks); Known issues reported from internal bug tracking or external information sharing resources (e.g., AI incident database, AVID, CVE, NVD, or OECD AI incident monitor); Human oversight roles and responsibilities.” A vendor that can produce something like this is a different category of vendor than one that can only produce a sales deck.

The reason to insist on this distinction — frontier-model provenance versus fine-tune/retrieval-layer provenance — is that NIST’s Generative AI Profile names exactly this failure mode as one of twelve defined generative-AI risks, Value Chain and Component Integration: “Non-transparent or untraceable integration of upstream third-party components, including data that has been improperly obtained or not processed and cleaned due to increased automation from GAI; improper supplier vetting across the AI lifecycle; or other issues that diminish transparency or accountability for downstream users.” “Downstream users” in that sentence is your organization. The risk NIST is describing is precisely a compliance team buying a SaaS product that turns out to have an unvetted, untraceable AI component several layers down — which is the scenario this guide exists to prevent.

Reviewing the Model Card or System Card

A model card is a short, structured document a model’s publisher produces describing what the model is, what it was trained to do, and where it’s known to fail. The practice traces to Margaret Mitchell and coauthors’ 2019 paper “Model Cards for Model Reporting,” and the format has since become close to a de facto standard: Hugging Face’s own model-card guidance for its model hub asks publishers to describe the model, its intended uses and limitations “including biases and ethical considerations,” the training parameters, which datasets were used to train it, and its evaluation results. Frontier labs now publish an equivalent document under the name “system card” for their flagship models.

A model card is not a substitute for a vendor’s own documentation, and asking for one is a cheap, high-signal test: a vendor that can point you to the specific model card or system card for the exact model version behind their product has done real diligence on their own supply chain. A vendor that can’t produce one, or produces a generic one that doesn’t match the model they’re actually using, is telling you something too. What to check on the card itself, specifically:

• Intended use and out-of-scope use. Does the vendor’s actual use case fall inside what the card says the model was evaluated for?
• Known limitations and failure modes. These map directly to the risk register entries your own AI risk assessment should be tracking for this vendor relationship.
• Evaluation results and their date. A card describing evaluations run against a model version the vendor stopped using eighteen months ago tells you nothing about what you’re buying today.
• The exact model version and deployment channel named on the card. This is where the card needs to match not just the model family the vendor claims to use, but the specific surface they access it through — see the NIKOLAI section below for why that distinction matters more than it looks.

Customer-Data Training Opt-Out: Get the Default in Writing

Whether your prompts, documents, and outputs get used to train the vendor’s (or the vendor’s vendor’s) next model is a default-setting question, and defaults vary by provider and change over time. The safest approach is procedural, not a matter of trusting a marketing claim: get the current default and the opt-out mechanism confirmed in the contract or order form, not just a support-page screenshot, because a vendor’s public policy can change and your signed terms are what actually protects you if it does.

A useful reference point for what a clear policy looks like: OpenAI’s current API documentation states plainly that “data sent to the OpenAI API is not used to train or improve OpenAI models” by default, and training only happens if a customer explicitly opts in to share data. Separately, API request/response content is retained for up to 30 days by default for abuse-monitoring purposes, though eligible customers can qualify for Zero Data Retention. Three separate facts are bundled in that one policy — default non-use for training, an explicit opt-in requirement to change that, and a separate retention window for a different purpose (abuse monitoring) — and a vendor building on top of an API like this should be able to tell you which of those three settings apply to your account specifically, and confirm it contractually.

None of this tells you what your specific vendor’s contract says. It tells you what a clearly-documented policy looks like, so you know what to ask for when your vendor’s documentation is vaguer than this.

AI-Output Indemnification: Read the Conditions, Not Just the Promise

An AI vendor offering to indemnify you against IP-infringement or other claims arising from AI-generated output sounds like a risk transfer. Often it is one — conditionally. The condition is the part compliance teams tend to skip, and it’s the part that determines whether the promise is worth anything when you actually need it.

Microsoft’s Customer Copyright Commitment for Azure OpenAI is a public, detailed example of how conditional this kind of commitment usually is. The commitment covers certain third-party IP claims over AI-generated output — but coverage is contingent on the customer having implemented a specific, named list of “Required Mitigations” at the time the output was generated, and being able to prove it: a content-filter configuration directing the model to avoid reproducing copyrighted material, and a retained testing/evaluation report showing the customer red-teamed or systematically tested for exactly this failure mode. Additional mitigations apply per use case — text generation and code generation each carry their own required filter settings. And the commitment doesn’t follow you if you bring your own model into the platform: “If Customer chooses to connect to or incorporate a model hosted outside of Copilot Studio, the Output Content from such model is not covered by the CCC unless such model runs in Azure OpenAI and meets the required mitigations on this page.”

The general lesson isn’t about Microsoft specifically — it’s that “we indemnify you for AI output” is not a single fact you can check off a list. It’s a conditional clause with its own compliance requirements, and those requirements usually include things your team has to actively do and document (specific safety-filter settings, retained evaluation records) rather than things the vendor does for you. When a vendor offers AI-output indemnification, the actual diligence question is: what do we have to configure and prove to keep this coverage, and who is responsible for maintaining that evidence?

Sub-Processor AI Disclosure

Most enterprise data-processing agreements already require the vendor (the processor) to disclose, and get your authorization for, any sub-processor it hands your data to — that obligation is standard enough that it’s written directly into data-protection law rather than left to individual contracts: the GDPR’s Article 28(2) states that “the processor shall not engage another processor without prior specific or general written authorisation of the controller,” and requires notice of any change so the controller has “the opportunity to object.” Most enterprise DPAs outside the EU borrow this same structure regardless of the jurisdiction they’re governed by.

The AI-specific extension of that same clause is straightforward and often missed: if the vendor’s product calls out to a frontier-model API to generate a result, that frontier lab is functionally a sub-processor of your data, whether or not the vendor’s sub-processor list names it that way. A sub-processor list that includes a cloud-hosting provider and a payment processor but doesn’t name the AI API the product actually calls is an incomplete list. The question to put to the vendor directly: does your sub-processor list include every AI/ML API our data is sent to, by name, and will you update it before adding or changing one? If the vendor can’t answer that specifically, the sub-processor list you were given during procurement should be treated as provisional, not final.

Audit Rights for AI-Specific Claims

A standard SaaS master service agreement typically gives you audit rights over security controls — access logs, encryption, SOC 2 scope. Those rights don’t automatically extend to AI-specific claims, because the claims are different in kind: “the model doesn’t retain our data for training,” “the model was evaluated for this failure mode,” “the indemnification mitigations were in place on the date of this output” are all claims about a system the vendor may not fully control (if they’re building on a third-party API) and about events in the past that need to be evidenced, not just asserted.

What to negotiate for specifically, beyond generic audit language: the right to request evidence that opt-out settings were actually configured as stated (not just a policy document, but confirmation for your account specifically); the right to request the evaluation or red-teaming records referenced in any indemnification clause, on a claim; and advance notice if the vendor changes which underlying model, model version, or deployment surface powers the feature you bought, since a model swap can silently change the risk profile of a product your own risk assessment already signed off on.

Where NIKOLAI Fits In

NIKOLAI, CASRAI’s own independent, unendorsed dictionary of frontier-AI-safety elements, defines an element in its N1 track (Actors, models and scope) called Deployment Surface: “The product, API or channel through which a model is made available (for example a consumer app, an enterprise API, a government cloud), distinct from deployment type.” It’s currently tagged proposed in NIKOLAI’s schema (release nikolai-v0.1).

That distinction — surface versus type — is exactly the gap a vendor-risk review needs to close and usually doesn’t. When you ask a SaaS vendor “which model powers this feature” and they answer “GPT” or “Claude,” they’ve told you the model family, not the surface you’re actually buying access to. NIKOLAI’s element page verifies this with xAI’s own model card for Grok 4, which names discrete surfaces as separate units of evaluation — “Grok 4 API,” described as “an enterprise use-focused API,” versus “Grok 4 Web,” the consumer-facing product — rather than treating “Grok 4” as one undifferentiated thing. Most other sources classify deployment type (consumer vs. enterprise) without going that granular on surface.

Why it matters for vendor risk specifically: Anthropic’s own Risk Report, cited on NIKOLAI’s Deployment Surface element page, discloses that “resolution times for these non-public jailbreaks have been as long as several months to reach all deployment surfaces on our most capable models.” Read plainly, that means a safety fix a frontier lab ships to its flagship consumer product can take months to reach the enterprise API surface a third-party vendor is embedding in the product you’re buying. A vendor’s claim that they use “the same model as ChatGPT” or “Claude, the same one everyone uses” is not, on its own, a claim about which surface, which patch level, or which safety-mitigation state you’re actually getting — and NIKOLAI’s own crosswalk on this element is a shadow mapping, CASRAI’s independent read of what each source discloses, not a confirmed or endorsed standard. The practical takeaway carries regardless: ask the vendor to name the specific surface and version, not just the model family, and ask when that surface last received the safety mitigations the frontier lab has published for its flagship version.

How This Maps to NIST AI RMF and ISO 42001

None of the questions above are invented for this guide. Two frameworks already covered on this site name third-party AI risk as a distinct governance obligation, and this checklist is a practical reading of both.

NIST’s AI Risk Management Framework devotes an entire GOVERN category to it — GOVERN 6, “Policies and procedures are in place to address AI risks and benefits arising from third-party software and data and other supply chain issues” — with two subcategories: GOVERN 6.1, “Policies and procedures are in place that address AI risks associated with third-party entities, including risks of infringement of a third-party’s intellectual property or other rights,” and GOVERN 6.2, “Contingency processes are in place to handle failures or incidents in third-party data or AI systems deemed to be high-risk.” See CASRAI’s guide to NIST’s Govern, Map, Measure, Manage functions for how GOVERN 6 fits alongside the framework’s other categories, and the Generative AI Profile guide for the full context around the Value Chain and Component Integration risk quoted above.

ISO/IEC 42001, the AI management system certification standard, addresses the same territory through a dedicated Annex A control: A.10, Third-party and customer relationships — “how responsibility for AI-related risk is allocated between the organisation and its suppliers, partners, and customers,” relevant, per CASRAI’s own certification guide, “wherever an organisation builds on a third-party model or provides AI capability to others.” An organization that has actually implemented A.10 as part of a 42001 certification has already built much of the documentation trail this guide asks you to request from a vendor. See CASRAI’s ISO/IEC 42001 certification guide for the full Annex A control list and what certification actually requires.

A Practical Pre-Procurement Sequence

1. Identify the actual model. Name, version, and whether it’s the vendor’s own model, a fine-tune, or a third-party API wrapper.
2. Request the model card or system card for that specific version, and check the date against the version you’re actually being sold.
3. Confirm the training-data default in writing — does your data train the vendor’s model, or any model it calls, by default, and what’s the opt-out mechanism.
4. Read the indemnification clause’s conditions, not just its headline — what does your team have to configure and document to keep coverage active.
5. Get the full sub-processor list, confirm it names every AI/ML API the product calls, and confirm you’ll be notified before it changes.
6. Negotiate audit rights specific to AI claims — opt-out configuration evidence, indemnification-mitigation evidence, and advance notice of model or surface changes.
7. Log the vendor relationship in your AI risk register as its own entry, with the model version and deployment surface as tracked fields, not folded into a generic “vendor risk” line item.

How This Differs From CASRAI’s Other AI Vendor-Adjacent Guides

Two other guides on this site sit close to this topic and are easy to confuse with it. AI Risk Assessment Framework and Risk Register covers how to build the register itself — the fields, the scoring method, the ownership model — for risks across your whole AI program; a vendor relationship is one possible entry in that register, but the guide doesn’t walk through how to vet a vendor before you sign. Third-Party AI Auditing, Explained is about the opposite direction of the same word “third-party”: it covers independent auditors assessing a lab’s or organization’s own AI system against a framework like ISO 42001 or the EU AI Act, a service you might commission after deployment. This guide is neither of those — it’s the specific, pre-signature work of a buyer deciding whether to trust a vendor’s AI-powered product in the first place.

Frequently Asked Questions

Is AI vendor risk assessment different from a normal SaaS vendor security review?
Yes. A conventional review checks the vendor’s own infrastructure and controls. AI vendor risk assessment adds a second layer: the behavior, training data, and safety posture of a model the vendor may not have built themselves, which a conventional security questionnaire has no fields for.

Do I need to see the underlying frontier model’s system card if my vendor just resells API access?
Yes, and this is the most commonly skipped step. If the vendor is a thin wrapper over a frontier lab’s API, the vendor’s own documentation usually can’t tell you what the model was trained on or how it was evaluated — only the frontier lab’s own model card or system card can, and the vendor should be able to point you directly to it for the specific version they use.

Does an AI-output indemnification clause protect us automatically?
Not usually. Published examples like Microsoft’s Customer Copyright Commitment for Azure OpenAI show these clauses are commonly conditional on the customer implementing and documenting specific safety mitigations. Read the conditions before treating the clause as coverage you already have.

What’s the single highest-signal question to ask an AI vendor during procurement?
Which specific model, version, and deployment surface (not just model family) powers this feature, and can you point us to its model card or system card. A vendor that can answer precisely has done real diligence on their own supply chain; a vendor that can only offer a general description usually hasn’t.

Where does this fit relative to NIST AI RMF and ISO 42001?
This guide is a practical checklist for implementing NIST’s GOVERN 6 (third-party AI policies) and ISO 42001’s Annex A.10 (third-party and customer relationships) at the specific moment of vendor procurement, before either framework’s broader governance program would otherwise touch the decision.

Why This Matters for Research Administration

University research computing and central-IT procurement offices are already running exactly this kind of review, often without a field for which model powers a licensed tool’s AI feature or which deployment surface a vendor is reselling access to. Many institutions’ data-use agreements and IRB protocols already restrict where identifiable research data can travel, and a vendor’s AI feature that silently routes prompts to a third-party frontier-model API can push data outside the boundary the DUA or IRB approval assumed — without procurement ever being told which API sits in the request path. A campus’s data-protection or research-computing office can add this guide’s exact sub-processor question — “does your sub-processor list include every AI/ML API our data is sent to, by name?” — straight into its existing research-tool vendor-risk process.