Skip to content
AI Engineer Guide.
midReportedcurated

Explain prefill versus decode in LLM inference. Which phase is compute-bound and which is memory-bandwidth-bound, and why does that matter for serving?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

What is chunked prefill and why does it help tail latency?
  • It splits a long prompt into pieces and interleaves those pieces with decode steps instead of running the whole prefill in one iteration.
  • Without it a large prompt occupies an iteration and every decoding request sees an inter-token latency spike.
  • The cost is scheduler complexity and a slightly slower prefill, and the gain is small when prompts are short and uniform.
  • It trades first-token latency on the long request for steadier streaming for everyone else.
Why does batching help decode far more than it helps prefill?
  • Each decode step reads the whole weight matrix for one token, so adding requests to the batch amortises that read across sequences instead of repeating it.
  • Prefill is already a large matrix multiply over the prompt, so it uses the device well without batching and gains less from more sequences.
  • The limit is cache memory rather than compute: every added sequence needs its own KV cache, so admission is a memory decision.
  • Past that point more concurrency means preemption and worse tails rather than more throughput.

Red flags

  • Says decode is compute-bound because it runs the model
  • Thinks TTFT is dominated by output length

Concepts this tests

Tags

Source: www.designgurus.io

More in LLM & transformer fundamentals