Skip to content
AI Engineer Guide.
midReportedcurated

Explain the KV cache. What does it store, why does it speed up generation, and how would you estimate its memory?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why can't you cache queries too?
  • A query is used once, for the token that produced it, and the next step needs a new query for the new token, so a stored query has no future reader.
  • Keys and values are different: later tokens read them again, which is what makes caching them a saving rather than a cost.
  • Caching queries would add memory and bandwidth without removing any work, which is the opposite of the cache's purpose.
What is prefix caching and when does it save money?
  • It reuses the keys and values of an identical prefix across requests, so the prefill for that prefix is skipped after the first request.
  • It pays off with a stable system prompt, tool definitions or a shared document, where many requests begin with the same bytes.
  • It requires byte-identical prefixes, so template discipline becomes a performance concern and any edit invalidates everything after the change.
  • It saves prefill compute and time-to-first-token, not decode: output tokens still cost a forward pass each.

Red flags

  • Says the cache stores activations for all layers
  • Cannot produce a per-token memory formula

Concepts this tests

Tags

Source: github.com

More in LLM & transformer fundamentals