staffPracticecurated
Your GPU has free KV-cache memory but throughput is poor and requests sit idle waiting for a batch to finish. What is the lever, and how does it differ from PagedAttention?
30-second answer
- The symptom is scheduling, not capacity: a static batch runs until its slowest sequence finishes, so freed slots sit idle while new requests wait.
- Continuous or iteration-level batching evicts finished sequences and admits new requests after every forward pass, keeping the batch full and the device busy.
- PagedAttention attacks a different problem: it stores KV in fixed-size blocks behind a block table, so memory is allocated on demand instead of reserved for the maximum length.
- PagedAttention raises how many sequences fit; continuous batching raises how well the resident sequences use the GPU, so the two compose rather than compete.
- The costs are scheduler and kernel complexity plus a fairness policy, because long generations otherwise starve short ones and prefill stalls spike tail latency.
Likely follow-ups
Open a question to rehearse the answer.
How do you combine paged allocation with continuous batching?
- Admission is decided from available cache blocks rather than a fixed batch size, so the scheduler cannot overcommit memory.
- Blocks are reference-counted so a shared prefix is read by many sequences, and copy-on-write handles the point where one diverges.
- Exhaustion needs an explicit policy: preempt, swap or reject, and account for the resulting recomputation in latency.
- Chunked prefill decides when a long prompt enters the schedule, which is the other half of keeping decode steady.
What metrics would show the scheduling fix worked?
- Time-to-first-token and inter-token latency at several concurrency levels, reported separately, because a single throughput number hides the trade.
- Queue wait by requested output length, where a rising floor is the signature of short requests starving behind long ones.
- Preemptions per request alongside cache utilisation, since over-admission looks like healthy throughput until it thrashes.
- Tokens per second alone is not evidence: it is an aggregate that can improve while every user waits longer.
Red flags
- Reaches for quantization when memory is not the constraint
- Treats PagedAttention and batching as the same feature
Concepts this tests
Tags
More in LLM & transformer fundamentals
- What is grouped-query attention, how does it differ from multi-head and multi-query attention, and what does it buy you?
- Explain the KV cache. What does it store, why does it speed up generation, and how would you estimate its memory?
- Explain prefill versus decode in LLM inference. Which phase is compute-bound and which is memory-bandwidth-bound, and why does that matter for serving?