Why it matters
An LLM application concatenates instructions and data into one channel, so text from a document can arrive in the same position as a system instruction. Debugging an injection report means finding how untrusted text reached the model, whether the model acted on it, and which capability made the action consequential. Teams often respond with prompt wording, which changes likelihood without changing the blast radius of a tool that can send email or run SQL.
Mental model
It is a mailroom where a letter can be written in the same handwriting as an internal memo, so the clerk cannot tell policy from post. The metaphor breaks because a mailroom has an identity check available, while the model has only position and wording to distinguish the two.
How it works
- untrusted · sourcesentry pointuntrustedRetrieved chunk
- entry pointuntrustedTool result
- entry pointuntrustedUser fileread as text
- neutral · prompt + turnRetrieved chunk· read as intentTool result· obeyedconcatenatedneutralOne token streamneededturn tool allowlist · open
- neutral · tools in the turnread-onlyneutralsearch_docs
- One token stream· externalturn tool allowlist · openexternal effectneutralsend_email
- One token stream· writeturn tool allowlist · openstate changeneutralrun_sql
entry points. List every path by which text the operator does not control reaches the prompt: retrieved chunks, tool observations, user files, web pages. Each is an entry point, not context.
Figure. The boundary is the tool set available in the turn, not the prompt wording. It passes a read-only tool and refuses an external effect or a write; it does not make the retrieved text trustworthy, and it does not stop a read-only tool from returning private data. The label-and-delimit step is not drawn as a gate because it raises cost rather than enforcing a boundary. Outbound validation and output scanning are not drawn. Shown with the whole registry reachable.
What to notice. Residual risk: the marker proves the boundary was crossed on this turn, not that every payload will be caught. Web pages arrive through the retrieved-chunk inlet; only three are drawn. Keep the payload in a regression set and run it after every prompt or tool change.
Text version
Three zones: UNTRUSTED · SOURCES holds a retrieved chunk, a tool result and a user file, each an entry point; NEUTRAL · PROMPT + TURN holds the single concatenated token stream; NEUTRAL · TOOLS IN THE TURN holds search_docs, send_email and run_sql. Three dashed red paths run from the entry points into the stream, labelled read as intent, obeyed and read as text. Three paths leave the stream: solid to the read-only search_docs, dashed red to send_email and run_sql. All three leaving edges carry the turn tool allowlist control: open, all three run; blocked, search_docs gets a teal tick and the external and write paths are cut at teal bars.
- Identify every path by which untrusted text reaches the prompt: retrieved chunks, tool results, user files, web pages.
- Trace the failing request to the exact text that carried the instruction and record where it entered.
- List the tools available in that turn and which of them can cause an external effect.
- Reproduce with a harmless marker, such as an instruction to output a fixed string, to confirm the boundary is crossed.
- Fix in layers: delimit and label untrusted content, restrict tool permissions, and validate outbound actions.
- Add the payload to a regression set and re-run it after every prompt or tool change.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Prompt delimiting | Cheap and immediate | Bypassed by determined input | The attacker controls document text |
| Tool allowlist | Bounds the blast radius | Blocks legitimate workflows | The agent legitimately needs broad access |
| Human confirmation | Stops consequential actions | Breaks automation and adds friction | The action is high volume |
| Output filtering | Catches known exfiltration patterns | False positives and an arms race | The payload is novel |
Good for
Cheap and immediate
Cost
Bypassed by determined input
Fails when
The attacker controls document text
- The attacker controls document text
- The agent legitimately needs broad access
- The action is high volume
- The payload is novel
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Prompt delimiting is good for Cheap and immediate, costs Bypassed by determined input, and fails when The attacker controls document text. Tool allowlist is good for Bounds the blast radius, costs Blocks legitimate workflows, and fails when The agent legitimately needs broad access. Human confirmation is good for Stops consequential actions, costs Breaks automation and adds friction, and fails when The action is high volume. Output filtering is good for Catches known exfiltration patterns, costs False positives and an arms race, and fails when The payload is novel
Failure modes
- Instruction in a document. A retrieved page tells the model to ignore prior instructions and the answer follows it. Debug: replay the request with the retrieved chunk replaced by a marker and see whether behaviour changes.
- Tool output trusted. A tool returns text containing an instruction and the next step obeys it. Debug: inspect the observation text in the trace for imperative language.
- Exfiltration through a link. The model embeds retrieved secrets in a URL the user is asked to open. Debug: scan generated output for external URLs containing encoded content.
In production
Log the source of every untrusted segment and the tools available per turn. Run an injection regression set in CI, and keep a kill switch for tools with external side effects.
Common misconceptions
People think: A strong system prompt prevents injection.
Actually: It raises the cost of a successful attack; it does not create a boundary between instructions and data.
People think: Injection only matters for chatbots.
Actually: It matters wherever untrusted text reaches a model with tools, which includes coding agents and support automation.
Interview angle
Start at the trust boundary, map capabilities, then describe layered mitigation and a reproduction. Weak answers propose a prompt phrase and never ask what the agent is allowed to do.
Sources
- OWASP Top 10 for LLM Applications — prompt injection and excessive agency as named risks.
- Greshake et al., Indirect Prompt Injection — how retrieved content becomes an attack vector.
- NIST AI Risk Management Framework — risk framing for capability and control decisions.