midReportedcurated
Explain prefill versus decode in LLM inference. Which phase is compute-bound and which is memory-bandwidth-bound, and why does that matter for serving?
30-second answer
- Prefill runs the whole prompt through the model in parallel, which is compute-bound and dominated by large matrix multiplies; it sets time-to-first-token, which grows with prompt length.
- Decode produces one token per forward pass and is memory-bandwidth-bound, because each step streams all weights and the KV cache from HBM while doing little arithmetic.
- Decode step time is dominated by the weight read, so adding a sequence to the batch costs far less than a proportional increase in step time until the batch saturates memory bandwidth.
- The split drives chunked prefill, continuous batching and prefill/decode disaggregation, because the two phases want different batch shapes and hardware ratios.
- Past saturation, step time and queueing both grow, so per-request latency and especially the tail worsen while aggregate tokens per second flattens; that is why TTFT and inter-token latency are tuned against each other.
Likely follow-ups
Open a question to rehearse the answer.
What is chunked prefill and why does it help tail latency?
- It splits a long prompt into pieces and interleaves those pieces with decode steps instead of running the whole prefill in one iteration.
- Without it a large prompt occupies an iteration and every decoding request sees an inter-token latency spike.
- The cost is scheduler complexity and a slightly slower prefill, and the gain is small when prompts are short and uniform.
- It trades first-token latency on the long request for steadier streaming for everyone else.
Why does batching help decode far more than it helps prefill?
- Each decode step reads the whole weight matrix for one token, so adding requests to the batch amortises that read across sequences instead of repeating it.
- Prefill is already a large matrix multiply over the prompt, so it uses the device well without batching and gains less from more sequences.
- The limit is cache memory rather than compute: every added sequence needs its own KV cache, so admission is a memory decision.
- Past that point more concurrency means preemption and worse tails rather than more throughput.
Red flags
- Says decode is compute-bound because it runs the model
- Thinks TTFT is dominated by output length
Concepts this tests
Tags
Source: www.designgurus.io
More in LLM & transformer fundamentals
- What is grouped-query attention, how does it differ from multi-head and multi-query attention, and what does it buy you?
- Explain the KV cache. What does it store, why does it speed up generation, and how would you estimate its memory?
- Explain speculative decoding. Why is it lossless, and when does it stop helping?