Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 13 of 24

Caching & invalidation

0 of 24 complete

Why it matters

Caching is the cheapest way to remove load and latency, and the reason it goes wrong is that the key and the invalidation rule are decided once and then the system changes around them. A new personalisation field, a permission check or a model version makes the stored copy wrong, and nothing fails loudly: users simply see yesterday's answer. Invalidation defines, per cached object, what makes it stale and who says so.

Mental model

It is a fridge with leftovers: the date label decides whether it is still good, and a smell is a late warning. The analogy breaks because a cache can be invalidated exactly when the source changes, while a fridge label is only ever an estimate.

How it works

Mechanism, step by stepInteractive
  1. 1Write the key from every input that changes the response: path, query, tenant, locale, permission scope, model or prompt version.
  2. 2Choose a freshness policy per object: a lifetime for tolerable staleness, validators for revalidation, explicit purge for correctness-critical data.
  3. 3Set the lifetime from how fast the source changes, and record the choice next to the cache definition.
  4. 4Invalidate by version rather than by deleting keys, so a deploy changes the key space and old entries age out.
  5. 5Add request coalescing or a short lock so a popular expired key does not cause a stampede.
  6. 6Serve stale on origin failure only if the product accepts it, and mark the response as stale.

Step 1 of 6

Write the key from every input that changes the response: path, query, tenant, locale, permission scope, model or prompt version.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Write the key from every input that changes the response: path, query, tenant, locale, pe… 2. Choose a freshness policy per object: a lifetime for tolerable staleness, validators for … 3. Set the lifetime from how fast the source changes, and record the choice next to the cach… 4. Invalidate by version rather than by deleting keys, so a deploy changes the key space and… 5. Add request coalescing or a short lock so a popular expired key does not cause a stampede. 6. Serve stale on origin failure only if the product accepts it, and mark the response as st…

  1. Write the key from every input that changes the response: path, query, tenant, locale, permission scope, model or prompt version.
  2. Choose a freshness policy per object: a lifetime for tolerable staleness, validators for revalidation, explicit purge for correctness-critical data.
  3. Set the lifetime from how fast the source changes, and record the choice next to the cache definition.
  4. Invalidate by version rather than by deleting keys, so a deploy changes the key space and old entries age out.
  5. Add request coalescing or a short lock so a popular expired key does not cause a stampede.
  6. Serve stale on origin failure only if the product accepts it, and mark the response as stale.

Trade-offs

ChoiceGood forCostFails when
Short lifetimeBounded staleness, simpleMore origin loadThe origin cannot absorb the miss rate
Explicit purgePrecise freshnessEvery writer must remember to purgeWrites happen in many services
Version in the keyAtomic invalidation, no purge callOld entries linger until evictionStorage is constrained
Cache everythingHighest hit rateStale and sensitive data risksResponses are personalised
Trade-off mapInteractive

Good for

Bounded staleness, simple

Cost

More origin load

Fails when

The origin cannot absorb the miss rate

  1. The origin cannot absorb the miss rate
  2. Writes happen in many services
  3. Storage is constrained
  4. Responses are personalised

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Short lifetime is good for Bounded staleness, simple, costs More origin load, and fails when The origin cannot absorb the miss rate. Explicit purge is good for Precise freshness, costs Every writer must remember to purge, and fails when Writes happen in many services. Version in the key is good for Atomic invalidation, no purge call, costs Old entries linger until eviction, and fails when Storage is constrained. Cache everything is good for Highest hit rate, costs Stale and sensitive data risks, and fails when Responses are personalised

Failure modes

  • Key missing a dimension. A user sees another locale's or tenant's content because the key omitted it. Debug: log the computed key and assert it changes when tenant, locale or permission scope changes.
  • Stampede on expiry. A hot key expires and the origin receives a burst of identical misses. Debug: chart miss rate against origin latency for a single key and look for a synchronised spike.
  • Purge that never fires. A writer path bypasses the invalidation call and stale data persists indefinitely. Debug: instrument writes and purges, and alert when a write has no corresponding purge.

In production

Track hit rate, age of served entries and origin load per key family. Alert on a sudden hit-rate drop, which usually means a key changed shape, and cap entry size so one object cannot evict the working set.

Common misconceptions

People think: Invalidation is only hard in distributed systems.

Actually: It is hard wherever more than one writer can change the source, which includes a single service with several code paths.

People think: A short lifetime removes the need for invalidation.

Actually: It bounds staleness but does not remove it, and the shorter the lifetime the more load the origin must absorb.

Interview angle

Begin with the key and the staleness budget, then choose between lifetime, validation and purge. Name the stampede and the writer that forgets to purge. Setting a cache header and moving on is not an invalidation design.

Sources