Why it matters
The same model served by two engines can differ by multiples in cost per token and in tail latency. Engines implement different schedulers, cache layouts and kernel paths, and those choices interact with your traffic shape. Choosing on a headline tokens-per-second number hides the behaviour that matters: what happens when a long prompt arrives during a burst of short ones.
Mental model
A serving engine is a restaurant kitchen. Batching is cooking several orders together; the KV cache is counter space. The metaphor breaks because orders cannot be cooked partially and set aside without holding memory, which is exactly what the cache does.
How it works
- 1Characterise traffic: input and output length distributions, concurrency, burstiness and acceptable p95.
- 2Pick the scheduling model: static batching for offline jobs, continuous batching for interactive traffic.
- 3Size KV-cache memory for the target concurrency and context length before tuning anything else.
- 4Choose quantisation from an accuracy test on your tasks, not from the format's reputation.
- 5Enable prefix caching if a large system prompt repeats across requests.
- 6Load-test with a realistic arrival process and report p50, p95 and goodput, not just peak throughput.
Step 1 of 6
Characterise traffic: input and output length distributions, concurrency, burstiness and acceptable p95.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Characterise traffic: input and output length distributions, concurrency, burstiness and … 2. Pick the scheduling model: static batching for offline jobs, continuous batching for inte… 3. Size KV-cache memory for the target concurrency and context length before tuning anything… 4. Choose quantisation from an accuracy test on your tasks, not from the format's reputation. 5. Enable prefix caching if a large system prompt repeats across requests. 6. Load-test with a realistic arrival process and report p50, p95 and goodput, not just peak…
- Characterise traffic: input and output length distributions, concurrency, burstiness and acceptable p95.
- Pick the scheduling model: static batching for offline jobs, continuous batching for interactive traffic.
- Size KV-cache memory for the target concurrency and context length before tuning anything else.
- Choose quantisation from an accuracy test on your tasks, not from the format's reputation.
- Enable prefix caching if a large system prompt repeats across requests.
- Load-test with a realistic arrival process and report p50, p95 and goodput, not just peak throughput.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Continuous batching | High GPU utilisation under mixed traffic | More complex scheduling and tail latency | Requests are uniform and latency-critical |
| Static batching | Predictable and simple | Idle capacity and poor burst handling | Traffic is interactive |
| Aggressive quantisation | Lower memory, more concurrency | Quality loss on reasoning or code | Output correctness is the product |
| Prefix caching | Large savings on shared system prompts | Memory held for cached prefixes | Prompts are unique per request |
Good for
High GPU utilisation under mixed traffic
Cost
More complex scheduling and tail latency
Fails when
Requests are uniform and latency-critical
- Requests are uniform and latency-critical
- Traffic is interactive
- Output correctness is the product
- Prompts are unique per request
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Continuous batching is good for High GPU utilisation under mixed traffic, costs More complex scheduling and tail latency, and fails when Requests are uniform and latency-critical. Static batching is good for Predictable and simple, costs Idle capacity and poor burst handling, and fails when Traffic is interactive. Aggressive quantisation is good for Lower memory, more concurrency, costs Quality loss on reasoning or code, and fails when Output correctness is the product. Prefix caching is good for Large savings on shared system prompts, costs Memory held for cached prefixes, and fails when Prompts are unique per request
Failure modes
- Cache exhaustion. Throughput collapses when concurrent long contexts exceed KV memory. Debug: monitor cache utilisation and reject or queue before eviction thrashing.
- Benchmark mismatch. Synthetic short prompts hide prefill cost for real documents. Debug: replay a sample of production length distributions.
- Quantisation regression. Accuracy drops only on the hardest slice. Debug: evaluate the quantised model on the labelled set before rollout.
In production
Track time-to-first-token, inter-token latency and goodput separately. Alert on cache utilisation and queue depth rather than GPU utilisation, which stays high while users wait.
Common misconceptions
People think: More GPU utilisation means better serving.
Actually: Utilisation can be high while requests queue; goodput and tail latency are the user-facing metrics.
People think: Quantisation is free performance.
Actually: It trades numerical precision for memory and speed, and the loss is task-dependent.
Interview angle
Start from the traffic distribution and the latency target, then explain batching and KV-cache sizing. Mention prefix caching and quantisation as conditional levers. A weak answer names an engine; a strong one explains which constraint chose it.
Sources
- Kwon et al., PagedAttention and vLLM — KV-cache paging and continuous batching.
- vLLM documentation — scheduler, prefix caching and quantisation options.
- NVIDIA Triton Inference Server documentation — a general-purpose serving surface with model ensembles.
Related
- Vector DBs compared
- Embedding models compared
- LLM providers compared