Why it matters
Many production queries repeat: support questions, policy lookups, near-identical paraphrases. Paying for retrieval and generation on each is expensive and slow. A cache-aside layer returns a stored answer on a hit and populates on a miss, but every hit is a bet that the new query deserved the old answer, and that bet is made by a similarity threshold rather than a human.
Mental model
A noticeboard of previous answers pinned beside the desk: fast to consult, and wrong when the new question is only similar to a pinned one. The metaphor breaks because the noticeboard cannot tell you how similar is similar enough; that threshold is a decision you must measure.
How it works
Step 1 of 5
Normalise and key
The key is composed before any lookup: tenant, prompt version, model and index version. Because the versions are in the key, a retired prompt cannot be served from cache.
Active layers
- Query
- Normalise + key
Hover a layer to read what it holds.
Next: Look up before paying
- Skips
- hit: retrieval + model
- Key
- tenant · versions
- Risk
- false hit · stampede
Figure. One request through the cache. The key carries tenant, prompt version, model and index version, so a retired prompt cannot be served from it. The dashed edge is a hit and skips retrieval and generation; the solid edge is a miss. Write-back is drawn below the answer because it is off the response path; threshold calibration is omitted.
What to notice. Both edges reach the same answer band and the user cannot tell which one was taken, which is the correctness problem rather than a rendering detail. The saving is real only on the dashed edge: a miss still pays for retrieval and the model call. A single-flight lock on the write-back is what stops a burst of misses for one key from becoming a burst of model calls.
Text version
A vertical stack. A normalised query is composed into a cache key of tenant, prompt version, model and index version, then looked up with a similarity threshold. A dashed lane labelled hit skips model runs from the lookup directly to the answer. A miss runs retrieval and generation, which produces the answer and a populate step; the write-back stores the result under the same key along a dash-dot lane back to the key band. Five steps cover key composition, lookup, the early return on a hit, the full cost of a miss, and asynchronous population with a single-flight lock.
- Start with exact-match caching on a normalised query plus tenant and prompt version, and measure the hit rate.
- If you add semantic matching, choose the threshold from labelled pairs of queries that should and should not share an answer.
- Include the index version, prompt version and model in the cache key, so a change invalidates by construction.
- Set a TTL by how fast the underlying facts change, not by storage cost.
- On a miss, populate asynchronously where possible, and use a single-flight lock to prevent a stampede.
- Log hits with their similarity score and review a sample, since a false hit is invisible to the user.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Exact-match cache | Safe, cheap, easy to invalidate | Low hit rate on paraphrase | Users phrase questions freely |
| Semantic cache with a high threshold | Few false hits | Small gain | Traffic is highly repetitive |
| Semantic cache with a low threshold | High hit rate | Wrong answers on near-misses | Answers depend on fine detail |
| Long TTL | Maximum savings | Stale answers after updates | The corpus changes often |
Good for
Safe, cheap, easy to invalidate
Cost
Low hit rate on paraphrase
Fails when
Users phrase questions freely
- Users phrase questions freely
- Traffic is highly repetitive
- Answers depend on fine detail
- The corpus changes often
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Exact-match cache is good for Safe, cheap, easy to invalidate, costs Low hit rate on paraphrase, and fails when Users phrase questions freely. Semantic cache with a high threshold is good for Few false hits, costs Small gain, and fails when Traffic is highly repetitive. Semantic cache with a low threshold is good for High hit rate, costs Wrong answers on near-misses, and fails when Answers depend on fine detail. Long TTL is good for Maximum savings, costs Stale answers after updates, and fails when The corpus changes often
Failure modes
- False hit. A similar question with a different constraint receives the wrong answer. Debug: review a sample of hits with similarity scores and the diff between the two queries.
- Stale after re-index. Documents change but cached answers persist. Debug: include the index version in the key and assert old-version entries are unreachable.
- Stampede on expiry. Many concurrent misses call the model at once. Debug: count model calls per key around a TTL boundary.
In production
Monitor hit rate, false-hit rate from sampled review, and cost per answered query. Invalidate by version rather than by hand, and cap the share of answers that come from cache.
Common misconceptions
People think: Semantic caching is a drop-in optimisation.
Actually: It changes the answer distribution and needs a labelled threshold, review and versioned invalidation.
People think: A higher hit rate is better.
Actually: A hit that answers the wrong question costs more than a miss, in trust if not in tokens.
Interview angle
Separate exact from semantic caching and state the correctness risk of the latter. Strong answers include key composition and invalidation. Weak ones quote a hit-rate target without saying how they would detect a wrong hit.
Sources
- Azure Architecture Center, Cache-Aside Pattern — load-on-miss, write-back and invalidation.
- Bang, GPTCache: An Open-Source Semantic Cache for LLM Applications — embedding-based similarity caching and its accuracy trade-off.
- Nishtala et al., Scaling Memcache at Facebook — cache consistency, stampedes and invalidation at scale.