Why it matters
Users ask the same thing in slightly different words, and each phrasing costs a retrieval plus a generation. Semantic caching turns that repetition into a lookup: embed the query, find a cached query within a threshold, return its answer. The saving is real, and so is the risk — near-identical questions can have different correct answers when the entity, the tenant or the date differs.
Mental model
Semantic caching is a FAQ pinned by the door: most visitors ask a question already answered there. The metaphor breaks because the pinned answer was written for a slightly different visitor, and the difference may be exactly what matters. The pin also never expires on its own, so it outlives the policy it describes.
How it works
- 1Decide what may be cached: stable, corpus-grounded answers that do not depend on the caller's personal data.
- 2Store the query embedding, the normalised query text, the answer, the cited document versions and a timestamp.
- 3Choose a similarity threshold from labelled paraphrase pairs, not from a default, and measure false-hit rate at that value.
- 4Scope the cache by tenant, permission set and locale so a hit cannot cross an access boundary.
- 5Invalidate entries when a cited document changes, when the model or prompt version changes, and on a TTL.
- 6Log every hit with the matched query and its similarity so a wrong answer can be traced to a false match.
Step 1 of 6
Decide what may be cached: stable, corpus-grounded answers that do not depend on the caller's personal data.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Decide what may be cached: stable, corpus-grounded answers that do not depend on the call… 2. Store the query embedding, the normalised query text, the answer, the cited document vers… 3. Choose a similarity threshold from labelled paraphrase pairs, not from a default, and mea… 4. Scope the cache by tenant, permission set and locale so a hit cannot cross an access boun… 5. Invalidate entries when a cited document changes, when the model or prompt version change… 6. Log every hit with the matched query and its similarity so a wrong answer can be traced t…
- Decide what may be cached: stable, corpus-grounded answers that do not depend on the caller's personal data.
- Store the query embedding, the normalised query text, the answer, the cited document versions and a timestamp.
- Choose a similarity threshold from labelled paraphrase pairs, not from a default, and measure false-hit rate at that value.
- Scope the cache by tenant, permission set and locale so a hit cannot cross an access boundary.
- Invalidate entries when a cited document changes, when the model or prompt version changes, and on a TTL.
- Log every hit with the matched query and its similarity so a wrong answer can be traced to a false match.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| High similarity threshold | Few wrong hits | Low hit rate and little saving | Users paraphrase heavily |
| Low threshold | High hit rate | Answers reused for different questions | Queries differ by an entity or a date |
| Long TTL | Maximum reuse | Stale answers after content changes | The corpus changes often |
| Global cache | Best hit rate | Cross-tenant leakage | Any content is permissioned |
Good for
Few wrong hits
Cost
Low hit rate and little saving
Fails when
Users paraphrase heavily
- Users paraphrase heavily
- Queries differ by an entity or a date
- The corpus changes often
- Any content is permissioned
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: High similarity threshold is good for Few wrong hits, costs Low hit rate and little saving, and fails when Users paraphrase heavily. Low threshold is good for High hit rate, costs Answers reused for different questions, and fails when Queries differ by an entity or a date. Long TTL is good for Maximum reuse, costs Stale answers after content changes, and fails when The corpus changes often. Global cache is good for Best hit rate, costs Cross-tenant leakage, and fails when Any content is permissioned
Failure modes
- False hit on a near-duplicate. A question about one plan returns the answer for another because the embeddings are close. Debug: review the matched query text and similarity for hits in the failing domain.
- Stale cached answer. The document changed but the cached answer still cites the old version. Debug: join cache entries to cited document versions and expire entries whose version is superseded.
- Cache crossing permissions. A user receives an answer generated for a broader scope. Debug: include the permission set in the cache key and test with two identities.
In production
Track hit rate, false-hit rate from sampled review, and age of served entries. Cap TTL by content volatility and purge the cache on every index swap.
Common misconceptions
People think: Semantic caching is a pure cost optimisation.
Actually: It changes correctness: a false hit returns a confidently wrong answer with no retrieval to catch it.
People think: A high similarity score means the same question.
Actually: Embeddings conflate near-duplicates that differ by the one token carrying the meaning.
Interview angle
Treat the threshold as a precision-recall decision, scope entries by permissions, and invalidate on content change. Name the near-duplicate entity case. Describing cached embeddings and cost savings without a false-hit measurement is the optimistic answer.
Sources
- RFC 9111, HTTP Caching — freshness, validation and invalidation semantics that any cache inherits.
- OpenAI, Prompt caching — exact-prefix reuse, the deterministic counterpart to semantic caching.
- Karpukhin et al., Dense Passage Retrieval — the embedding similarity that a semantic cache threshold relies on.