seniorReportedcurated
Your transformer's KV cache grows too large during long sequence generation and you run out of memory. Walk me through diagnosis and mitigation.
30-second answer
- Size the cache first: 2 × layers × kv_heads × head_dim × tokens × batch × bytes, because it scales linearly with both context and batch and often exceeds the weights.
- Then check allocation: fragmentation and buffers reserved for the maximum length waste memory that the arithmetic says should be free.
- Levers on per-token bytes are GQA or MQA to cut KV heads and FP8 or INT8 KV quantisation; levers on allocation are paged blocks and prefix caching.
- Levers on demand are beam-width reduction, a lower context cap and a smaller batch, all of which trade throughput or capability for memory.
- Aggressive KV quantisation costs quality on long-context recall and smaller batches cost throughput, so measure time-to-first-token, inter-token latency and task evals after each change.
Likely follow-ups
Open a question to rehearse the answer.
Which of these levers changes per-token bytes versus allocation efficiency?
- GQA, MQA and KV quantisation change bytes per token: fewer KV heads or fewer bits per element shrink the cache for every sequence.
- Paged allocation and prefix caching change how memory is used: blocks remove max-length reservation, and shared prefixes stop duplicated storage.
- Beam width and batch size change how many caches exist at once, which is demand rather than per-token cost.
- Confusing the two leads to buying GPUs when the real fix is a block allocator.
How would you decide between shrinking context and shrinking batch?
- Compute what each buys: cache bytes scale with tokens times batch, so halving either halves the cache but changes a different product promise.
- Shrinking batch protects answer quality and context but lowers throughput and raises queueing, which shows up first in time-to-first-token.
- Shrinking context protects throughput and concurrency but truncates long inputs, which shows up as missing evidence rather than as an error.
- Decide from the SLO you promised and the p95 context length you serve, then verify with task evals rather than memory numbers alone.
Red flags
- Says buy more GPUs without sizing the cache
- Confuses weight memory with KV-cache memory
Concepts this tests
Tags
Source: github.com
More in LLM & transformer fundamentals
- What is grouped-query attention, how does it differ from multi-head and multi-query attention, and what does it buy you?
- Explain the KV cache. What does it store, why does it speed up generation, and how would you estimate its memory?
- Explain prefill versus decode in LLM inference. Which phase is compute-bound and which is memory-bandwidth-bound, and why does that matter for serving?