Why it matters
A single request uses a fraction of the hardware, so serving one at a time wastes capacity. Batching amortises weights and memory reads across sequences, raising throughput. But every added request lengthens the batch, and a request that joins late waits for the current step. The engineering problem is finding a batch size and admission policy that keeps the GPU busy without letting queue time dominate p95 latency.
Mental model
Batching is a bus that leaves when full: more passengers per trip, but each waits for the others. The metaphor breaks because the bus can pick up passengers mid-route, and a late passenger can still slow the whole trip, so the timetable is a policy rather than a schedule.
How it works
- 1Measure single-request latency and throughput first, so the batching gain has a baseline.
- 2Choose a serving mode: continuous batching for chat traffic, static batching for offline jobs with similar lengths.
- 3Set a maximum batch size from memory, then tune a maximum wait time so small batches do not idle the GPU.
- 4Add admission control in front of the queue, with a per-tenant concurrency cap and a bounded queue length.
- 5Measure p50, p95 and p99 separately, since queueing shows up in the tail long before the median moves.
- 6Load-test at the target rate with realistic prompt and output length distributions, not fixed short prompts.
Step 1 of 6
Measure single-request latency and throughput first, so the batching gain has a baseline.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Measure single-request latency and throughput first, so the batching gain has a baseline. 2. Choose a serving mode: continuous batching for chat traffic, static batching for offline … 3. Set a maximum batch size from memory, then tune a maximum wait time so small batches do n… 4. Add admission control in front of the queue, with a per-tenant concurrency cap and a boun… 5. Measure p50, p95 and p99 separately, since queueing shows up in the tail long before the … 6. Load-test at the target rate with realistic prompt and output length distributions, not f…
- Measure single-request latency and throughput first, so the batching gain has a baseline.
- Choose a serving mode: continuous batching for chat traffic, static batching for offline jobs with similar lengths.
- Set a maximum batch size from memory, then tune a maximum wait time so small batches do not idle the GPU.
- Add admission control in front of the queue, with a per-tenant concurrency cap and a bounded queue length.
- Measure p50, p95 and p99 separately, since queueing shows up in the tail long before the median moves.
- Load-test at the target rate with realistic prompt and output length distributions, not fixed short prompts.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Large batches | High throughput, low cost per token | Longer per-request latency | The workload is interactive |
| Small batches | Low latency | Underused hardware | Traffic is steady and cost matters |
| Continuous batching | Good utilisation with mixed lengths | More complex serving stack | Requests have very different output lengths and priorities |
| Unbounded queue | Absorbs bursts | Latency grows without limit under overload | Traffic exceeds capacity for minutes |
Good for
High throughput, low cost per token
Cost
Longer per-request latency
Fails when
The workload is interactive
- The workload is interactive
- Traffic is steady and cost matters
- Requests have very different output lengths and priorities
- Traffic exceeds capacity for minutes
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Large batches is good for High throughput, low cost per token, costs Longer per-request latency, and fails when The workload is interactive. Small batches is good for Low latency, costs Underused hardware, and fails when Traffic is steady and cost matters. Continuous batching is good for Good utilisation with mixed lengths, costs More complex serving stack, and fails when Requests have very different output lengths and priorities. Unbounded queue is good for Absorbs bursts, costs Latency grows without limit under overload, and fails when Traffic exceeds capacity for minutes
Failure modes
- Queue collapse. Latency rises, clients retry, and the queue grows faster than it drains. Debug: plot arrival rate against completion rate and check retry volume during the spike.
- Batch padding waste. Mixed sequence lengths pad to the longest and waste compute. Debug: log batch composition and compare tokens processed with tokens billed.
- Fairness starvation. One tenant's long requests fill every batch and others time out. Debug: measure per-tenant latency under a shared load test.
In production
Alert on queue depth and p95 time to first token rather than average latency. Cap concurrency per tenant and shed load explicitly when the queue passes a threshold.
Common misconceptions
People think: Bigger batches always lower cost.
Actually: They raise throughput until memory or latency limits bind, after which they add queueing delay without savings.
People think: Concurrency is free if the provider autoscales.
Actually: Autoscaling reacts to load with a delay, so a burst still queues; client-side limits are what keep the system stable.
Interview angle
Explain the throughput-latency curve, name the serving technique that refills batches, and say where queueing appears in the metrics. Strong answers include admission control and per-tenant fairness. Weak answers say batching improves performance without distinguishing throughput from tail latency.
Sources
- Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — chunked prefill and continuous batching under a latency target.
- Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention — KV cache memory as the constraint on batch size.
- Zhong et al., DistServe — separating prefill and decode to meet latency targets.