Prompt injection

Prompt injection is an attack in which untrusted content — crafted directly by an attacker, or planted in data the model later retrieves, such as a web page, document, or email — causes an AI system to follow instructions its operator never intended, because the underlying model cannot reliably distinguish trusted instructions from untrusted data in its context window. Greshake et al. (2023) formalised the distinction between direct prompt injection, where a user attacks the system through the input they control, and indirect prompt injection, where the malicious payload arrives through data the model processes on the user’s behalf — a retrieved document, a webpage, a tool result — with no direct attacker interaction with the victim at all.

Why indirect injection is the harder problem

Indirect prompt injection is OWASP’s top-ranked risk for large-language-model applications specifically because it scales without requiring the attacker to interact with the target: poisoning one webpage or document that a retrieval-augmented or tool-using agent is likely to fetch can compromise every user session that touches it. This makes it a supply-chain-style risk for AI systems, closer to a watering-hole attack than to a traditional direct exploit.

How this differs from jailbreaking

Jailbreaking targets the model’s own trained behaviour — convincing the model itself to set aside the safety training or content policy it was aligned to follow. Prompt injection targets the application built around the model: it exploits the fact that instructions and untrusted data share the same channel (the context window), regardless of whether the model’s own alignment is intact. A well-aligned model can still be prompt-injected if the application feeding it text doesn’t separate instructions from retrieved content.

Common defences

• Privilege separation between the system prompt/instructions and untrusted retrieved or user-supplied content, so the model is told which is which.
• Output filtering and constrained action spaces for tool-using or agentic systems, so an injected instruction cannot trigger a consequential action unchecked.
• Human-in-the-loop confirmation before high-stakes actions (sending data externally, executing code, making purchases).
• Input sanitisation and content provenance checks on retrieved documents.

No defence is currently considered complete; red-teaming for injection resistance is standard practice ahead of deploying retrieval-augmented or tool-using systems, precisely because the failure mode is adversarial and adaptive rather than something a fixed test suite reliably catches.

References

• Greshake et al., ‘Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection’ (arXiv, 2023); OWASP Top 10 for LLM Applications.