Why it matters
A long system prompt, tool catalogue or document set is identical on every request, yet a stateless API recomputes its attention state each time. That prefill is the expensive part of the first token. Caching keeps the computed state for a prefix and reuses it, turning a repeated long prompt into a short incremental one. The constraint is exactness: one changed character early in the prefix makes the cached state unusable.
Mental model
Prompt caching is keeping a prepared base stock rather than chopping vegetables for every order. The image breaks because the stock must match the recipe exactly: one different ingredient at the start and the whole batch is useless, while the kitchen still pays for whatever is added at the end.
How it works
Request layout · illustrative tokens
6,300 input+ 500 reserve= 6,800 of 8,000
1,200 free
Step 1Request 1 — write
No cached state exists yet. The provider computes the 5,400-token prefix and stores it at the cache boundary.
Suffix · sent either way
Output reserve
500Pick a section to read its token count, its role and its state.
Figure. A stable prefix is computed once and read back on the next request. The cache boundary is drawn at 5,400 tokens; one changed byte before it turns the whole prefix into a miss, while the suffix is sent either way.
What to notice. Cache lifetime, pricing and boundary mechanics are provider-specific. This figure shows prefix reuse and invalidation only — not a billed cost, a measured latency or a claim about any named provider.
Text version
An 8,000-token window with a 500-token output reserve. A vertical cache boundary is drawn at 5,400 tokens, 67.5% of the window. Left of it is the prefix: system prompt 1,000, tool definitions 2,000 and reference documents 2,400, marked in teal on a hit. Right of it is the suffix, 900 tokens of user question and retrieved passages, bracketed and labelled as sent on every request. The default first step writes the prefix and labels the boundary as a write. The second step labels it a hit while the suffix is processed. A switch labelled System prompt toggles between unchanged and edited; on the repeat-request steps, edited labels the boundary a miss and hatches the three prefix blocks in red. The third step is the edit test, and the fourth places variable content after the boundary. Selecting any block shows its count, its side of the boundary, and its cache state.
- Identify the stable prefix: system instructions, tool definitions and reference documents that do not change between requests.
- Order the prompt so everything variable, such as user text and retrieved chunks, comes after the stable part.
- Mark the cache boundary where the provider requires it, or rely on automatic prefix matching where it does not.
- Keep the prefix byte-identical across requests, including whitespace and serialisation order.
- Measure hit rate and time to first token before and after enabling caching.
- Re-evaluate after any prompt edit, since edits inside the prefix reset the cache for every request.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Long cached prefix | Cheap repeated prefill, low first-token latency | Wasted tokens when the cache misses | Requests share little context |
| Short prompt, no cache | Simple, no invalidation logic | Full prefill every call | The prefix is long and stable |
| Cache per user | Personal context reused | Many small caches, low hit rate | Each user has a different long prompt |
| Cache shared across users | High hit rate | Requires identical prefix and careful data separation | The prefix contains per-user data |
Good for
Cheap repeated prefill, low first-token latency
Cost
Wasted tokens when the cache misses
Fails when
Requests share little context
- Requests share little context
- The prefix is long and stable
- Each user has a different long prompt
- The prefix contains per-user data
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Long cached prefix is good for Cheap repeated prefill, low first-token latency, costs Wasted tokens when the cache misses, and fails when Requests share little context. Short prompt, no cache is good for Simple, no invalidation logic, costs Full prefill every call, and fails when The prefix is long and stable. Cache per user is good for Personal context reused, costs Many small caches, low hit rate, and fails when Each user has a different long prompt. Cache shared across users is good for High hit rate, costs Requires identical prefix and careful data separation, and fails when The prefix contains per-user data
Failure modes
- Silent invalidation. A timestamp or request id at the top of the prompt changes every call and the cache never hits. Debug: hash the prefix per request and check that the hash is stable across calls.
- Cache miss billed as normal. Cost stays high because writes are priced above reads on providers that separate them, and the prefix is rarely reused. Debug: compare cache read and write token counts in the usage fields.
- Stale content. A cached document set is reused after the source changed. Debug: version the prefix and invalidate on document update rather than on time alone.
In production
Monitor cache hit rate and effective cost per request, and alert when a prompt deploy drops the hit rate. Keep variable content strictly after the cache boundary.
Common misconceptions
People think: Caching reduces output cost.
Actually: It reduces the cost and latency of processing the input prefix; generated tokens are still billed normally.
People think: Any repeated prompt benefits.
Actually: Only an exact prefix match does; near-duplicates and reordered sections miss.
Interview angle
Explain what is cached, which is the attention state for a prefix, and what breaks a hit. Strong answers cover ordering, hit-rate monitoring and the cost of a miss. Weak answers describe caching as memoising the response and cannot say why a one-character edit invalidates the whole prefix.
Sources
- Gim et al., Prompt Cache: Modular Attention Reuse for Low-Latency Inference — reusing attention states across prompts with shared segments.
- Anthropic, Prompt caching — cache breakpoints, lifetime and usage reporting.
- Zheng et al., SGLang — radix-tree prefix sharing in a serving system.