Why it matters
A single request alternates between two workloads with opposite hardware profiles. Prefill is a large matrix multiplication over many tokens, which uses the arithmetic units well. Decode is a sequence of skinny matrix-vector products, which leaves the arithmetic units idle while waiting on memory. Systems that treat inference as one uniform operation mis-size batches and report throughput that no interactive user experiences.
Mental model
Prefill is reading a whole brief before answering; decode is writing the reply one word at a time while re-reading notes. The metaphor breaks because the notes are not re-read from the page — the key-value cache is held in memory, so the cost is moving that memory through the chip, not fetching it from storage.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.
- pairs per pass
- 4 × 4 = 16
- passes
- 1
- pairs total
- 16
- rows live
- 4/4
Step 1 of 3
Prefill · all four queries
The prompt is processed in one batched pass. Every query reads every key, including the future keys that the causal mask removes afterwards, because the score grid is computed before the mask is applied.
Next: Chunked prefill · two queries
Figure. Four queries against four keys, illustrative counts. A cell prints 1 when that query-key pair is computed in the pass and 0 when it is not. Prefill computes all 16 pairs in one pass; chunked prefill computes two rows per pass; decode computes one row per step, so the same 16 pairs arrive in four skinny passes. The grid is work per pass, not visible context: a prefill cell above the diagonal is still computed, then removed by the causal mask.
What to notice. The arithmetic is a schedule, not a saving: 4 × 4 = 16 pairs in one pass, 2 × 4 = 8 per pass over two passes, and 1 × 4 = 4 per pass over four. The total is identical. What changes is the shape each pass presents to the array, which is why prefill fills a compute-bound matmul while decode issues a skinny matrix-vector product and waits on the cache. Row totals count pairs computed per query, column totals count the active queries per key. The example omits the cache reads and the mask itself, and the key count grows in a real decode step, so the 1 × 4 row widens; a real server interleaves several requests, and a decode row may be batched with others.
Text version
Three frames of a four by four query-key grid. Rows are queries q1 to q4 and columns keys k1 to k4; 1 means the pair is computed in this pass. The first frame prints 1 in all sixteen cells, row totals 4 and column totals 4. The second prints 1 in the first two rows only, row totals 4, 4, 0, 0 and column totals 2. The third prints 1 in the fourth row only, row totals 0, 0, 0, 4 and column totals 1. Steps compare one pass of 16 pairs, two passes of 8 and four passes of 4; the readout reports the query, the key and whether the pair is computed.
- Tokenise the prompt and run all its positions through the model in one batched forward pass, storing keys and values for every layer.
- Compute the first output token from the final prompt position and return it as soon as it is ready, which defines time to first token.
- For each subsequent token, run a forward pass with a single query position against the cached keys and values.
- Append the new key and value to the cache and continue until an end token or a length cap is reached.
- Batch decode steps across requests with continuous batching so one long generation does not block short ones.
- Split long prefills into chunks and interleave them with decode steps to keep inter-token latency bounded.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Large prefill batches | High throughput on long prompts | Longer time to first token per request | Users are waiting interactively |
| Chunked prefill | Smoother inter-token latency under load | More scheduling complexity and some overhead | Prompts are short and uniform |
| Continuous batching | High device utilisation across mixed requests | Per-request latency is harder to predict | Requests must be strictly isolated |
| Single-request serving | Predictable latency, simple accounting | Very low utilisation and high cost per token | Traffic is more than a trickle |
Good for
High throughput on long prompts
Cost
Longer time to first token per request
Fails when
Users are waiting interactively
- Users are waiting interactively
- Prompts are short and uniform
- Requests must be strictly isolated
- Traffic is more than a trickle
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Large prefill batches is good for High throughput on long prompts, costs Longer time to first token per request, and fails when Users are waiting interactively. Chunked prefill is good for Smoother inter-token latency under load, costs More scheduling complexity and some overhead, and fails when Prompts are short and uniform. Continuous batching is good for High device utilisation across mixed requests, costs Per-request latency is harder to predict, and fails when Requests must be strictly isolated. Single-request serving is good for Predictable latency, simple accounting, costs Very low utilisation and high cost per token, and fails when Traffic is more than a trickle
Failure modes
- Prefill and decode mixed in one metric. Average latency looks acceptable while interactive users see long waits. Debug: report time to first token and inter-token latency as separate series.
- Cache growth unaccounted. Decode slows as context grows because each step reads a larger cache. Debug: plot per-step latency against current sequence length for one request.
- Head-of-line blocking. A long prefill delays every decode step in the batch. Debug: correlate inter-token latency spikes with arrivals of long prompts.
In production
Alert on time to first token and inter-token latency separately, and on queue depth by phase. Cap maximum output length and prompt length explicitly, since both determine the worst-case latency you are willing to serve.
Common misconceptions
People think: Inference speed is one number.
Actually: Prefill and decode have different bottlenecks, so tokens per second measured on prompt processing says little about generation latency.
People think: Batching always reduces latency.
Actually: Batching raises throughput but can increase per-request latency by making each step larger; whether that is acceptable depends on the latency target.
Interview angle
Frame the two phases by what they compute and what binds them, then map those to time to first token and inter-token latency. Explain why decode is bandwidth-bound and how batching and chunked prefill shift the trade-off. Weak answers describe generation as a loop without recognising that the prompt pass has a different hardware profile.
Sources
- Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention — prefill and decode scheduling with a paged KV cache.
- Dao et al., FlashAttention — why the prefill pass is memory-traffic limited.
- vLLM documentation — continuous batching and chunked prefill in a production server.