Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 56 of 57

Semantic caching

0 of 57 complete

Why it matters

Users ask the same thing in slightly different words, and each phrasing costs a retrieval plus a generation. Semantic caching turns that repetition into a lookup: embed the query, find a cached query within a threshold, return its answer. The saving is real, and so is the risk — near-identical questions can have different correct answers when the entity, the tenant or the date differs.

Mental model

Semantic caching is a FAQ pinned by the door: most visitors ask a question already answered there. The metaphor breaks because the pinned answer was written for a slightly different visitor, and the difference may be exactly what matters. The pin also never expires on its own, so it outlives the policy it describes.

How it works

Mechanism, step by stepInteractive
  1. 1Decide what may be cached: stable, corpus-grounded answers that do not depend on the caller's personal data.
  2. 2Store the query embedding, the normalised query text, the answer, the cited document versions and a timestamp.
  3. 3Choose a similarity threshold from labelled paraphrase pairs, not from a default, and measure false-hit rate at that value.
  4. 4Scope the cache by tenant, permission set and locale so a hit cannot cross an access boundary.
  5. 5Invalidate entries when a cited document changes, when the model or prompt version changes, and on a TTL.
  6. 6Log every hit with the matched query and its similarity so a wrong answer can be traced to a false match.

Step 1 of 6

Decide what may be cached: stable, corpus-grounded answers that do not depend on the caller's personal data.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Decide what may be cached: stable, corpus-grounded answers that do not depend on the call… 2. Store the query embedding, the normalised query text, the answer, the cited document vers… 3. Choose a similarity threshold from labelled paraphrase pairs, not from a default, and mea… 4. Scope the cache by tenant, permission set and locale so a hit cannot cross an access boun… 5. Invalidate entries when a cited document changes, when the model or prompt version change… 6. Log every hit with the matched query and its similarity so a wrong answer can be traced t…

  1. Decide what may be cached: stable, corpus-grounded answers that do not depend on the caller's personal data.
  2. Store the query embedding, the normalised query text, the answer, the cited document versions and a timestamp.
  3. Choose a similarity threshold from labelled paraphrase pairs, not from a default, and measure false-hit rate at that value.
  4. Scope the cache by tenant, permission set and locale so a hit cannot cross an access boundary.
  5. Invalidate entries when a cited document changes, when the model or prompt version changes, and on a TTL.
  6. Log every hit with the matched query and its similarity so a wrong answer can be traced to a false match.

Trade-offs

ChoiceGood forCostFails when
High similarity thresholdFew wrong hitsLow hit rate and little savingUsers paraphrase heavily
Low thresholdHigh hit rateAnswers reused for different questionsQueries differ by an entity or a date
Long TTLMaximum reuseStale answers after content changesThe corpus changes often
Global cacheBest hit rateCross-tenant leakageAny content is permissioned
Trade-off mapInteractive

Good for

Few wrong hits

Cost

Low hit rate and little saving

Fails when

Users paraphrase heavily

  1. Users paraphrase heavily
  2. Queries differ by an entity or a date
  3. The corpus changes often
  4. Any content is permissioned

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: High similarity threshold is good for Few wrong hits, costs Low hit rate and little saving, and fails when Users paraphrase heavily. Low threshold is good for High hit rate, costs Answers reused for different questions, and fails when Queries differ by an entity or a date. Long TTL is good for Maximum reuse, costs Stale answers after content changes, and fails when The corpus changes often. Global cache is good for Best hit rate, costs Cross-tenant leakage, and fails when Any content is permissioned

Failure modes

  • False hit on a near-duplicate. A question about one plan returns the answer for another because the embeddings are close. Debug: review the matched query text and similarity for hits in the failing domain.
  • Stale cached answer. The document changed but the cached answer still cites the old version. Debug: join cache entries to cited document versions and expire entries whose version is superseded.
  • Cache crossing permissions. A user receives an answer generated for a broader scope. Debug: include the permission set in the cache key and test with two identities.

In production

Track hit rate, false-hit rate from sampled review, and age of served entries. Cap TTL by content volatility and purge the cache on every index swap.

Common misconceptions

People think: Semantic caching is a pure cost optimisation.

Actually: It changes correctness: a false hit returns a confidently wrong answer with no retrieval to catch it.

People think: A high similarity score means the same question.

Actually: Embeddings conflate near-duplicates that differ by the one token carrying the meaning.

Interview angle

Treat the threshold as a precision-recall decision, scope entries by permissions, and invalidate on content change. Name the near-duplicate entity case. Describing cached embeddings and cost savings without a false-hit measurement is the optimistic answer.

Sources