staffReportedcurated
An agent reads retrieved documents and tool outputs and can take actions. How do you defend against indirect prompt injection, and why can't a better system prompt solve it?
30-second answer
- Instructions and untrusted data share one token channel, so a retrieved document or tool output saying to ignore previous instructions can redirect the model, and no system prompt removes that possibility.
- Defend in layers: delimit and label retrieved text as data, give the model least-privilege tools scoped to the task, keep secrets out of context, and require human approval for irreversible or external actions.
- Validate tool arguments against a schema and an allowlist, check outputs before they reach a system of record, and log the decision so a blocked action is explainable.
- The lethal trifecta is private data, untrusted content and an outbound channel in one context; remove any leg and the attack loses its payoff.
- Every restriction reduces utility, adds latency and produces false positives, so scope permissions to the task and measure attack success alongside task success on benign inputs.
Likely follow-ups
Open a question to rehearse the answer.
What is the lethal trifecta and how would you break it?
- An agent that reads support tickets, can see customer records and can send email has all three legs, so a planted ticket can exfiltrate data.
- Break it by removing the egress tool from the reading context, or by requiring approval and an allowlisted recipient for any send.
- Scope credentials per user and tenant, because a shared mailbox token turns a prompt-level mistake into a company-wide capability.
How would you test injection defenses before launch?
- Run an injection suite that plants payloads in documents, tool outputs and image text, and measure attack success on poisoned inputs beside task success on benign ones.
- Keep every successful payload as a regression case, and add a canary document whose contents should never appear in an outbound request.
- Test the deterministic layer directly, because a prompt change can look safe while the permission check is the only thing still blocking the action.
Red flags
- Believes a strong system prompt is sufficient
- Gives the agent broad credentials and relies on model judgment
Concepts this tests
Tags
Source: github.com
More in Prompting & context engineering
- You stuff 60 retrieved documents into one prompt and the model uses facts from the beginning and end but ignores a crucial fact in the middle. Diagnose it and fix it.
- Your JSON parsing works in development but fails intermittently in production even at temperature 0. Diagnose it and describe the durable fix.
- Your chain-of-thought prompt is not improving accuracy on a reasoning task. What do you check and what do you change?