Skip to content
AI Engineer Guide.
staffPracticecurated

Your GPU has free KV-cache memory but throughput is poor and requests sit idle waiting for a batch to finish. What is the lever, and how does it differ from PagedAttention?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How do you combine paged allocation with continuous batching?
  • Admission is decided from available cache blocks rather than a fixed batch size, so the scheduler cannot overcommit memory.
  • Blocks are reference-counted so a shared prefix is read by many sequences, and copy-on-write handles the point where one diverges.
  • Exhaustion needs an explicit policy: preempt, swap or reject, and account for the resulting recomputation in latency.
  • Chunked prefill decides when a long prompt enters the schedule, which is the other half of keeping decode steady.
What metrics would show the scheduling fix worked?
  • Time-to-first-token and inter-token latency at several concurrency levels, reported separately, because a single throughput number hides the trade.
  • Queue wait by requested output length, where a rising floor is the signature of short requests starving behind long ones.
  • Preemptions per request alongside cache utilisation, since over-admission looks like healthy throughput until it thrashes.
  • Tokens per second alone is not evidence: it is an aggregate that can improve while every user waits longer.

Red flags

  • Reaches for quantization when memory is not the constraint
  • Treats PagedAttention and batching as the same feature

Concepts this tests

Tags

More in LLM & transformer fundamentals