Skip to content
AI Engineer Guide.
seniorReportedcurated

Your transformer's KV cache grows too large during long sequence generation and you run out of memory. Walk me through diagnosis and mitigation.

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Which of these levers changes per-token bytes versus allocation efficiency?
  • GQA, MQA and KV quantisation change bytes per token: fewer KV heads or fewer bits per element shrink the cache for every sequence.
  • Paged allocation and prefix caching change how memory is used: blocks remove max-length reservation, and shared prefixes stop duplicated storage.
  • Beam width and batch size change how many caches exist at once, which is demand rather than per-token cost.
  • Confusing the two leads to buying GPUs when the real fix is a block allocator.
How would you decide between shrinking context and shrinking batch?
  • Compute what each buys: cache bytes scale with tokens times batch, so halving either halves the cache but changes a different product promise.
  • Shrinking batch protects answer quality and context but lowers throughput and raises queueing, which shows up first in time-to-first-token.
  • Shrinking context protects throughput and concurrency but truncates long inputs, which shows up as missing evidence rather than as an error.
  • Decide from the SLO you promised and the p95 context length you serve, then verify with task evals rather than memory numbers alone.

Red flags

  • Says buy more GPUs without sizing the cache
  • Confuses weight memory with KV-cache memory

Concepts this tests

Tags

Source: github.com

More in LLM & transformer fundamentals