Why it matters
Static batching wastes the device whenever sequences finish at different times: the batch runs until its longest member completes, and short requests sit in a slot that could have been reused. Continuous batching fixes the utilisation problem by making the batch a moving set, but it moves the complexity into the scheduler, which must now decide admission, preemption and prefill placement on every step while keeping per-request latency acceptable.
Mental model
It is a bus that picks up passengers at every stop instead of running express from the depot. The bus metaphor breaks because seats are memory blocks, and a passenger who boards with a very long journey occupies a seat that a short-trip passenger could have used twice.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Aggregate tokens per second against batch size for continuous and static scheduling, with the inter-token latency of the same batches below. Both panels come from one model — a decode step emits one token per sequence, so tokens per second is batch divided by step time — and the constants are illustrative.
What to notice. From batch 1 to 32 continuous scheduling lifts throughput from 92 to about 873 tokens a second while per-token latency rises from 11 to 37 ms. Throughput still reaches about 1,010 at batch 64, but the 55 ms interactive budget is crossed near batch 54, past the 48-sequence cache bound: the last sixteen sequences add about 5% throughput and 27% latency. Static batching sits at about half the continuous curve because finished sequences hold their slots.
Text version
Top panel: tokens per second against batch size from 1 to 64. The continuous curve rises from 92 and flattens toward 1,200, reaching about 1,010 at batch 64; the dashed static curve ends near 525. A vertical cache-bound line sits at 48 sequences. Bottom panel: inter-token latency rising from 11 ms to about 63 ms, with a horizontal 55 ms interactive budget crossed near batch 54. A batch-size slider moves a cursor and reads throughput, the static comparison and latency at once.
- Characterise output length variance first, because continuous batching pays off exactly where lengths diverge.
- Set the admission rule from available cache blocks rather than a fixed batch size, so the scheduler cannot overcommit memory.
- Choose how prompt processing enters the schedule: separate iterations, or chunked prefill interleaved with decode.
- Define a fairness policy for long generations, such as a token cap per request per window or priority for short jobs.
- Handle preemption explicitly: recompute or swap the evicted sequence and account for that work in latency.
- Report time-to-first-token and inter-token latency separately at several concurrency levels, since a single throughput number hides the trade.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Admit every iteration | Highest device utilisation | Long prompts delay other requests' tokens | Tail latency is a product requirement |
| Chunked prefill | Smoother inter-token latency under mixed traffic | More scheduler complexity and slightly slower prefill | Prompts are short and uniform |
| Large batch target | More tokens per second overall | Each sequence holds cache longer | Memory is the binding constraint |
| Priority scheduling | Protects interactive traffic | Batch work is delayed unpredictably | All traffic shares one latency objective |
Good for
Highest device utilisation
Cost
Long prompts delay other requests' tokens
Fails when
Tail latency is a product requirement
- Tail latency is a product requirement
- Prompts are short and uniform
- Memory is the binding constraint
- All traffic shares one latency objective
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Admit every iteration is good for Highest device utilisation, costs Long prompts delay other requests' tokens, and fails when Tail latency is a product requirement. Chunked prefill is good for Smoother inter-token latency under mixed traffic, costs More scheduler complexity and slightly slower prefill, and fails when Prompts are short and uniform. Large batch target is good for More tokens per second overall, costs Each sequence holds cache longer, and fails when Memory is the binding constraint. Priority scheduling is good for Protects interactive traffic, costs Batch work is delayed unpredictably, and fails when All traffic shares one latency objective
Failure modes
- Starvation of short requests. A few long generations hold the batch and short requests wait behind them. Debug: plot queue wait by requested output length and look for a rising floor.
- Prefill stalls. A large prompt occupies an iteration and every decoding request sees a latency spike. Debug: correlate inter-token latency spikes with prompt length in the trace.
- Over-admission. The scheduler accepts more sequences than the cache can hold and preemption thrashes. Debug: log preemptions per request alongside cache utilisation.
In production
Watch queue wait, preemption rate and cache utilisation together; a healthy throughput chart can hide a queue that grows all afternoon. Cap maximum context per request so admission stays predictable under mixed traffic.
Common misconceptions
People think: Continuous batching only improves throughput.
Actually: It improves device utilisation, and it can worsen tail latency unless prefill placement and fairness are designed deliberately.
People think: A bigger batch is always better.
Actually: Batch size is bounded by cache memory and by the latency you promised; past that point more concurrency means preemption, not progress.
Interview angle
Explain iteration-level scheduling, then the two constraints that make it hard: cache memory and prefill interference. A strong answer describes the fairness policy and the metric pair used to judge it. Weak answers say it keeps the device busy and stop there.
Sources
- Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models — iteration-level scheduling for generative serving.
- Kwon et al., PagedAttention — block-based cache management that makes continuous batching practical.