Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 17 of 33

Debug: injection

0 of 33 complete

Why it matters

An LLM application concatenates instructions and data into one channel, so text from a document can arrive in the same position as a system instruction. Debugging an injection report means finding how untrusted text reached the model, whether the model acted on it, and which capability made the action consequential. Teams often respond with prompt wording, which changes likelihood without changing the blast radius of a tool that can send email or run SQL.

Mental model

It is a mailroom where a letter can be written in the same handwriting as an internal memo, so the clerk cannot tell policy from post. The metaphor breaks because a mailroom has an identity check available, while the model has only position and wording to distinguish the two.

How it works

Blast radius: untrusted text, one stream, three toolsInteractive
turn tool allowlist
Step 1/6 · entry points
  1. untrusted · sources
    entry pointuntrusted
    Retrieved chunk
  2. entry pointuntrusted
    Tool result
  3. entry pointuntrusted
    User file
    read as text
  4. neutral · prompt + turn
    Retrieved chunk· read as intent
    Tool result· obeyed
    concatenatedneutral
    One token stream
    neededturn tool allowlist · open
  5. neutral · tools in the turn
    read-onlyneutral
    search_docs
  6. One token stream· externalturn tool allowlist · open
    external effectneutral
    send_email
  7. One token stream· writeturn tool allowlist · open
    state changeneutral
    run_sql
trusted pathtainted pathcontrol openboundary holdspassespending approvaltrusted zoneuntrusted zoneneutral zone3 zones · 7 nodes · illustrative geometry

entry points. List every path by which text the operator does not control reaches the prompt: retrieved chunks, tool observations, user files, web pages. Each is an entry point, not context.

Figure. The boundary is the tool set available in the turn, not the prompt wording. It passes a read-only tool and refuses an external effect or a write; it does not make the retrieved text trustworthy, and it does not stop a read-only tool from returning private data. The label-and-delimit step is not drawn as a gate because it raises cost rather than enforcing a boundary. Outbound validation and output scanning are not drawn. Shown with the whole registry reachable.

What to notice. Residual risk: the marker proves the boundary was crossed on this turn, not that every payload will be caught. Web pages arrive through the retrieved-chunk inlet; only three are drawn. Keep the payload in a regression set and run it after every prompt or tool change.

Text version

Three zones: UNTRUSTED · SOURCES holds a retrieved chunk, a tool result and a user file, each an entry point; NEUTRAL · PROMPT + TURN holds the single concatenated token stream; NEUTRAL · TOOLS IN THE TURN holds search_docs, send_email and run_sql. Three dashed red paths run from the entry points into the stream, labelled read as intent, obeyed and read as text. Three paths leave the stream: solid to the read-only search_docs, dashed red to send_email and run_sql. All three leaving edges carry the turn tool allowlist control: open, all three run; blocked, search_docs gets a teal tick and the external and write paths are cut at teal bars.

  1. Identify every path by which untrusted text reaches the prompt: retrieved chunks, tool results, user files, web pages.
  2. Trace the failing request to the exact text that carried the instruction and record where it entered.
  3. List the tools available in that turn and which of them can cause an external effect.
  4. Reproduce with a harmless marker, such as an instruction to output a fixed string, to confirm the boundary is crossed.
  5. Fix in layers: delimit and label untrusted content, restrict tool permissions, and validate outbound actions.
  6. Add the payload to a regression set and re-run it after every prompt or tool change.

Trade-offs

ChoiceGood forCostFails when
Prompt delimitingCheap and immediateBypassed by determined inputThe attacker controls document text
Tool allowlistBounds the blast radiusBlocks legitimate workflowsThe agent legitimately needs broad access
Human confirmationStops consequential actionsBreaks automation and adds frictionThe action is high volume
Output filteringCatches known exfiltration patternsFalse positives and an arms raceThe payload is novel
Trade-off mapInteractive

Good for

Cheap and immediate

Cost

Bypassed by determined input

Fails when

The attacker controls document text

  1. The attacker controls document text
  2. The agent legitimately needs broad access
  3. The action is high volume
  4. The payload is novel

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Prompt delimiting is good for Cheap and immediate, costs Bypassed by determined input, and fails when The attacker controls document text. Tool allowlist is good for Bounds the blast radius, costs Blocks legitimate workflows, and fails when The agent legitimately needs broad access. Human confirmation is good for Stops consequential actions, costs Breaks automation and adds friction, and fails when The action is high volume. Output filtering is good for Catches known exfiltration patterns, costs False positives and an arms race, and fails when The payload is novel

Failure modes

  • Instruction in a document. A retrieved page tells the model to ignore prior instructions and the answer follows it. Debug: replay the request with the retrieved chunk replaced by a marker and see whether behaviour changes.
  • Tool output trusted. A tool returns text containing an instruction and the next step obeys it. Debug: inspect the observation text in the trace for imperative language.
  • Exfiltration through a link. The model embeds retrieved secrets in a URL the user is asked to open. Debug: scan generated output for external URLs containing encoded content.

In production

Log the source of every untrusted segment and the tools available per turn. Run an injection regression set in CI, and keep a kill switch for tools with external side effects.

Common misconceptions

People think: A strong system prompt prevents injection.

Actually: It raises the cost of a successful attack; it does not create a boundary between instructions and data.

People think: Injection only matters for chatbots.

Actually: It matters wherever untrusted text reaches a model with tools, which includes coding agents and support automation.

Interview angle

Start at the trust boundary, map capabilities, then describe layered mitigation and a reproduction. Weak answers propose a prompt phrase and never ask what the agent is allowed to do.

Sources