Written and maintained by CASRAI Editorial Board
Last updated
Data poisoning is an attack in which an adversary tampers with the data a machine-learning model learns from, so that the trained model behaves badly. NIST defines poisoning attacks broadly as “adversarial attacks during the training stage of the ML algorithm” (NIST AI 100-2 E2025, section 2.3). This page is for researchers, data stewards and security staff who curate or consume training data and need to know what the attack types are, where in the pipeline they enter, and which controls the two main reference documents recommend.
What data poisoning is
A model’s behaviour is shaped by its training data. If an attacker can alter, insert or relabel part of that data, they can degrade the model’s accuracy, plant a hidden trigger, or bias its outputs. The OWASP Top 10 for LLM Applications 2025 (entry LLM04, Data and Model Poisoning) describes it as manipulation of training data so a model learns harmful behaviour, introducing “vulnerabilities, backdoors, or biases”. NIST notes that data poisoning applies to all learning paradigms, and that the attack can operate at scale: an adversary with limited resources could control a fraction of the public datasets that models are trained on.
How it differs from evasion and prompt injection
- Poisoning acts at the training stage, before the model is deployed. The damage is built into the model.
- Evasion acts at deployment: the attacker perturbs an input to a model that is already trained.
- Prompt injection manipulates a deployed language model through its input text, not through its training data.
The distinction matters for controls: poisoning is addressed mainly through data governance and training-pipeline integrity, the other two mainly at run time. For how the security framing relates to the safety framing, see AI safety vs AI security.
Types of poisoning attack (NIST taxonomy)
NIST AI 100-2 E2025 (Vassilev, Oprea, Fordyce, Anderson, Davies and Hamin, March 2025; doi 10.6028/NIST.AI.100-2e2025) groups poisoning attacks by the violation they cause. Availability poisoning “typically cause[s] indiscriminate degradation of the ML model on all samples”, whereas targeted and backdoor poisoning “induce integrity violations on a small set of target samples”. The report’s attacker-capability categories are training data control, model (parameter) control, label control, source code control and test data control.
| Type | Effect | What the attacker must control | NIST identifier |
|---|---|---|---|
| Availability poisoning (section 2.3.1) | Indiscriminate degradation of the model | Training data | NISTAML.013 (Availability Poisoning) |
| Targeted poisoning (2.3.2) | Integrity violation on a small set of target samples | Training data | NISTAML.024 (Targeted Poisoning) |
| Backdoor poisoning (2.3.3) | Model misbehaves when a trigger pattern is present | Training data and test data (a pattern added to poisoned and test samples) | NISTAML.023 (Backdoor Poisoning); NISTAML.021 (Clean-label Backdoor) |
| Model poisoning (2.3.4) | Malicious changes to the model itself | Model parameters or training code; most prevalent in federated learning | NISTAML.011 (Model Poisoning); NISTAML.026 |
NIST observes that backdoor attacks are the only attacks in the literature requiring both training-data and test-data control. Clean-label poisoning (NISTAML.012) is a related technique in which the poisoned samples keep labels that look correct. Each NIST subsection also discusses mitigations and their limitations; the report is the authoritative place to check the current status of any specific technique.
Where it enters the pipeline (OWASP lifecycle stages)
OWASP LLM04 identifies three stages at which poisoning can occur: pre-training, fine-tuning and embedding. It also notes that a backdoor “may leave the model’s behavior untouched until a certain trigger causes it to change”, which is why ordinary accuracy testing can miss it, and that malicious pickling of model files can run code when a model is loaded.
| Stage | What is exposed |
|---|---|
| Pre-training | Large scraped or public corpora |
| Fine-tuning | Task-specific or vendor-supplied datasets |
| Embedding | Documents converted to vectors for retrieval |
| Model loading | Downloaded model files (malicious pickling can run code at load, per OWASP) |
The OWASP scenarios include biased or misinformation outputs, toxic data that was not filtered, falsified documents, inadequate filtering, and a backdoor trigger.
Defences and a provenance checklist for research groups
The OWASP mitigation list for LLM04 translates into a checklist for any group that assembles training data:
- Track data origin and transformations. OWASP recommends tooling such as CycloneDX and an ML bill of materials. See data provenance and the term training-data provenance.
- Vet data vendors and validate their outputs against trusted sources.
- Version your data (OWASP names data version control, DVC) so a change can be detected and rolled back.
- Apply infrastructure controls: sandboxing and anomaly detection on ingestion so the model is not trained on unvetted sources.
- Monitor training: track training loss and set thresholds that flag unusual behaviour.
- Red-team the trained model for backdoor and bias behaviour; see AI red teaming explained.
- Ground outputs at inference with retrieval (RAG) as a secondary control.
For risk-management framing, the NIST AI RMF Generative AI Profile sits alongside these documents, and the comparison NIST AI 100-2 vs OWASP LLM Top 10 explains how the two taxonomies map to each other.
FAQ
What is data poisoning in machine learning?
It is the deliberate manipulation of training data so that the resulting model degrades, contains a hidden trigger, or produces biased outputs. NIST places it in the training-stage attack family.
What is the difference between data poisoning and model poisoning?
Data poisoning alters the data the model learns from. Model poisoning alters the model or its training process directly; NIST says it is most prevalent in federated learning, while data poisoning applies to all learning paradigms.
What is a backdoor attack?
A backdoor attack makes the model behave normally until an input contains a trigger pattern. NIST notes it needs control over both training and test data.
Can poisoning be detected by testing accuracy?
Not reliably for backdoors, because OWASP notes the model may behave normally until the trigger appears. Provenance tracking, data versioning and targeted red-teaming address the gap.
Sources
- NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, March 2025: csrc.nist.gov/pubs/ai/100/2/e2025/final (read 2026-11-08).
- OWASP Top 10 for LLM Applications 2025, LLM04 Data and Model Poisoning: genai.owasp.org (read 2026-11-08).
