midReportedcurated
Explain the KV cache. What does it store, why does it speed up generation, and how would you estimate its memory?
30-second answer
- During decode each new token needs the keys and values of every previous token at every layer, and those vectors do not change once computed.
- The cache stores them, so each step projects only the new token and attends over cached tensors instead of recomputing the prefix, turning quadratic decode work into linear work per token.
- Memory is 2 × layers × kv_heads × head_dim × tokens × bytes, so a 70B model with 80 layers, 8 KV heads and head_dim 128 at two bytes per element is about 0.31 MB per token and roughly 10 GB at 32k tokens.
- Queries are not cached because each query is used once; only K and V are reused, which is also why GQA and MQA shrink the cache by sharing K/V heads.
- The trade is memory capacity and bandwidth against compute: at long context and high concurrency the cache, not the weights, caps how many sequences fit.
Likely follow-ups
Open a question to rehearse the answer.
Why can't you cache queries too?
- A query is used once, for the token that produced it, and the next step needs a new query for the new token, so a stored query has no future reader.
- Keys and values are different: later tokens read them again, which is what makes caching them a saving rather than a cost.
- Caching queries would add memory and bandwidth without removing any work, which is the opposite of the cache's purpose.
What is prefix caching and when does it save money?
- It reuses the keys and values of an identical prefix across requests, so the prefill for that prefix is skipped after the first request.
- It pays off with a stable system prompt, tool definitions or a shared document, where many requests begin with the same bytes.
- It requires byte-identical prefixes, so template discipline becomes a performance concern and any edit invalidates everything after the change.
- It saves prefill compute and time-to-first-token, not decode: output tokens still cost a forward pass each.
Red flags
- Says the cache stores activations for all layers
- Cannot produce a per-token memory formula
Concepts this tests
Tags
Source: github.com
More in LLM & transformer fundamentals
- What is grouped-query attention, how does it differ from multi-head and multi-query attention, and what does it buy you?
- Explain prefill versus decode in LLM inference. Which phase is compute-bound and which is memory-bandwidth-bound, and why does that matter for serving?
- Explain speculative decoding. Why is it lossless, and when does it stop helping?