Why it matters
Many production requests repeat: the same support question, the same classification of the same document, the same system prompt with a fixed user message. Paying to regenerate an identical answer is waste, and on latency-sensitive paths a hit is far cheaper than a call. The difficulty is that a completion depends on more than the visible text, so a key that omits the model version or the retrieved context returns an answer that was correct under different conditions.
Mental model
It is a pantry of finished dishes: if the order is identical, you plate what is already made. The analogy breaks with semantic caching, where the cook decides that a similar order counts as the same dish, and that judgement is where wrong answers enter.
How it works
- 1Normalise the request into a canonical key: model revision, template version, rendered variables, sampling parameters and tool configuration.
- 2Scope the key by tenant and by any authorisation context that changes what the answer may contain.
- 3Choose a time to live from how fast the underlying facts change, not from how long the cache is convenient.
- 4For semantic caching, embed the request, set a similarity threshold, and validate the threshold on labelled paraphrase and non-paraphrase pairs.
- 5Bypass the cache for personalised, time-sensitive or side-effecting requests, and mark cached responses in the payload.
- 6Track hit rate, false-hit rate and cost avoided separately from model quality metrics.
Step 1 of 6
Normalise the request into a canonical key: model revision, template version, rendered variables, sampling parameters and tool configuration.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Normalise the request into a canonical key: model revision, template version, rendered va… 2. Scope the key by tenant and by any authorisation context that changes what the answer may… 3. Choose a time to live from how fast the underlying facts change, not from how long the ca… 4. For semantic caching, embed the request, set a similarity threshold, and validate the thr… 5. Bypass the cache for personalised, time-sensitive or side-effecting requests, and mark ca… 6. Track hit rate, false-hit rate and cost avoided separately from model quality metrics.
- Normalise the request into a canonical key: model revision, template version, rendered variables, sampling parameters and tool configuration.
- Scope the key by tenant and by any authorisation context that changes what the answer may contain.
- Choose a time to live from how fast the underlying facts change, not from how long the cache is convenient.
- For semantic caching, embed the request, set a similarity threshold, and validate the threshold on labelled paraphrase and non-paraphrase pairs.
- Bypass the cache for personalised, time-sensitive or side-effecting requests, and mark cached responses in the payload.
- Track hit rate, false-hit rate and cost avoided separately from model quality metrics.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Exact-match cache | Safe, cheap, easy to reason about | Low hit rate on natural variation | Users phrase the same question differently |
| Semantic cache | Higher hit rate on paraphrase | Wrong answers on near-misses | The domain has many similar but distinct questions |
| Long time to live | High hit rate | Stale answers after a data change | Facts change often |
| Per-tenant cache | No cross-customer leakage | Lower hit rate and more storage | Traffic is dominated by shared public questions |
Good for
Safe, cheap, easy to reason about
Cost
Low hit rate on natural variation
Fails when
Users phrase the same question differently
- Users phrase the same question differently
- The domain has many similar but distinct questions
- Facts change often
- Traffic is dominated by shared public questions
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Exact-match cache is good for Safe, cheap, easy to reason about, costs Low hit rate on natural variation, and fails when Users phrase the same question differently. Semantic cache is good for Higher hit rate on paraphrase, costs Wrong answers on near-misses, and fails when The domain has many similar but distinct questions. Long time to live is good for High hit rate, costs Stale answers after a data change, and fails when Facts change often. Per-tenant cache is good for No cross-customer leakage, costs Lower hit rate and more storage, and fails when Traffic is dominated by shared public questions
Failure modes
- Stale after a prompt change. The template is edited but the key does not include its version, so old answers persist. Debug: assert that the key hash changes when the template changes.
- Semantic false hit. A similar-sounding but different question receives the wrong answer. Debug: sample cache hits and have a reviewer label them for correctness weekly.
- Cross-tenant leak. A shared cache returns an answer built from another customer's context. Debug: search stored entries for foreign tenant identifiers and fail the build if the key omits tenant scope.
In production
Monitor hit rate, false-hit rate and entry age; a rising hit rate with falling quality is the signature of an over-eager threshold. Cap entry size and lifetime so the cache cannot become a second copy of your data with weaker access control.
Common misconceptions
People think: Caching is free performance.
Actually: Every cache adds a correctness surface: stale entries, wrong keys and leakage, all invisible until a user sees the wrong answer.
People think: A high similarity score means the answers are interchangeable.
Actually: Similar questions can require different answers; the threshold has to be validated on labelled pairs, not chosen by intuition.
Interview angle
Begin with the key, because the key is where correctness lives, then cover scoping, lifetime and invalidation. Describe how a semantic threshold would be validated. Treating caching as a pure latency win, with no mention of staleness or leakage, is the weak answer.
Sources
- RFC 9111, HTTP Caching — freshness, validation and cache key semantics.
- Bang, GPTCache — semantic caching for model responses and its accuracy trade-off.
- OpenAI, Prompt caching — provider-side prefix caching and what invalidates it.