Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 2 of 3

Serving engines compared

0 of 3 complete

Why it matters

The same model served by two engines can differ by multiples in cost per token and in tail latency. Engines implement different schedulers, cache layouts and kernel paths, and those choices interact with your traffic shape. Choosing on a headline tokens-per-second number hides the behaviour that matters: what happens when a long prompt arrives during a burst of short ones.

Mental model

A serving engine is a restaurant kitchen. Batching is cooking several orders together; the KV cache is counter space. The metaphor breaks because orders cannot be cooked partially and set aside without holding memory, which is exactly what the cache does.

How it works

Mechanism, step by stepInteractive
  1. 1Characterise traffic: input and output length distributions, concurrency, burstiness and acceptable p95.
  2. 2Pick the scheduling model: static batching for offline jobs, continuous batching for interactive traffic.
  3. 3Size KV-cache memory for the target concurrency and context length before tuning anything else.
  4. 4Choose quantisation from an accuracy test on your tasks, not from the format's reputation.
  5. 5Enable prefix caching if a large system prompt repeats across requests.
  6. 6Load-test with a realistic arrival process and report p50, p95 and goodput, not just peak throughput.

Step 1 of 6

Characterise traffic: input and output length distributions, concurrency, burstiness and acceptable p95.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Characterise traffic: input and output length distributions, concurrency, burstiness and … 2. Pick the scheduling model: static batching for offline jobs, continuous batching for inte… 3. Size KV-cache memory for the target concurrency and context length before tuning anything… 4. Choose quantisation from an accuracy test on your tasks, not from the format's reputation. 5. Enable prefix caching if a large system prompt repeats across requests. 6. Load-test with a realistic arrival process and report p50, p95 and goodput, not just peak…

  1. Characterise traffic: input and output length distributions, concurrency, burstiness and acceptable p95.
  2. Pick the scheduling model: static batching for offline jobs, continuous batching for interactive traffic.
  3. Size KV-cache memory for the target concurrency and context length before tuning anything else.
  4. Choose quantisation from an accuracy test on your tasks, not from the format's reputation.
  5. Enable prefix caching if a large system prompt repeats across requests.
  6. Load-test with a realistic arrival process and report p50, p95 and goodput, not just peak throughput.

Trade-offs

ChoiceGood forCostFails when
Continuous batchingHigh GPU utilisation under mixed trafficMore complex scheduling and tail latencyRequests are uniform and latency-critical
Static batchingPredictable and simpleIdle capacity and poor burst handlingTraffic is interactive
Aggressive quantisationLower memory, more concurrencyQuality loss on reasoning or codeOutput correctness is the product
Prefix cachingLarge savings on shared system promptsMemory held for cached prefixesPrompts are unique per request
Trade-off mapInteractive

Good for

High GPU utilisation under mixed traffic

Cost

More complex scheduling and tail latency

Fails when

Requests are uniform and latency-critical

  1. Requests are uniform and latency-critical
  2. Traffic is interactive
  3. Output correctness is the product
  4. Prompts are unique per request

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Continuous batching is good for High GPU utilisation under mixed traffic, costs More complex scheduling and tail latency, and fails when Requests are uniform and latency-critical. Static batching is good for Predictable and simple, costs Idle capacity and poor burst handling, and fails when Traffic is interactive. Aggressive quantisation is good for Lower memory, more concurrency, costs Quality loss on reasoning or code, and fails when Output correctness is the product. Prefix caching is good for Large savings on shared system prompts, costs Memory held for cached prefixes, and fails when Prompts are unique per request

Failure modes

  • Cache exhaustion. Throughput collapses when concurrent long contexts exceed KV memory. Debug: monitor cache utilisation and reject or queue before eviction thrashing.
  • Benchmark mismatch. Synthetic short prompts hide prefill cost for real documents. Debug: replay a sample of production length distributions.
  • Quantisation regression. Accuracy drops only on the hardest slice. Debug: evaluate the quantised model on the labelled set before rollout.

In production

Track time-to-first-token, inter-token latency and goodput separately. Alert on cache utilisation and queue depth rather than GPU utilisation, which stays high while users wait.

Common misconceptions

People think: More GPU utilisation means better serving.

Actually: Utilisation can be high while requests queue; goodput and tail latency are the user-facing metrics.

People think: Quantisation is free performance.

Actually: It trades numerical precision for memory and speed, and the loss is task-dependent.

Interview angle

Start from the traffic distribution and the latency target, then explain batching and KV-cache sizing. Mention prefix caching and quantisation as conditional levers. A weak answer names an engine; a strong one explains which constraint chose it.

Sources