seniorReportedcurated
Explain speculative decoding. Why is it lossless, and when does it stop helping?
30-second answer
- A small draft model proposes k tokens, and the target scores all k+1 positions in one forward pass instead of k sequential passes.
- Each proposal is accepted with probability min(1, p_target/p_draft), and on the first rejection the decoder resamples from a corrected residual distribution.
- That rejection rule makes the output distribution exactly the target's, so it is lossless rather than an approximation that trades quality for speed.
- The gain comes from decode being bandwidth-bound: verifying five tokens costs about what generating one costs, and typical wins are 1.5–3×.
- It stops helping at high batch, where spare compute for drafting disappears, or when acceptance is low, where every rejected proposal is wasted verification.
- It also adds a draft model's weights to memory and reduces cache headroom, which can lower concurrency.
Likely follow-ups
Open a question to rehearse the answer.
Why does the benefit shrink at high batch size?
- Speculation spends idle compute on drafting, and at high batch there is no idle compute left because the device is already saturated with real requests.
- The draft's proposals also compete for the same memory and bandwidth, so the cost shows up as lower throughput rather than lower latency.
- Acceptance rate alone does not predict the win, because the comparison has to be end-to-end latency at the batch sizes you actually serve.
- That is why it is rolled out per workload slice against a control path and disabled when it stops paying.
How would you measure the gain in a live deployment?
- Track acceptance rate and verified tokens per request by request type, because code, structured output and prose accept at very different rates.
- Compare end-to-end latency against a non-speculative control path at the same batch size rather than against a historical baseline.
- Watch maximum concurrent sequences with and without the draft loaded, since extra weights reduce cache headroom.
- Add a distribution test on a fixed prompt set to catch a broken verification rule that changes outputs subtly.
Red flags
- Calls it an approximation that trades quality for speed
- Ignores the high-batch regime where it hurts
Concepts this tests
Tags
Source: github.com
More in LLM & transformer fundamentals
- What is grouped-query attention, how does it differ from multi-head and multi-query attention, and what does it buy you?
- Explain the KV cache. What does it store, why does it speed up generation, and how would you estimate its memory?
- Explain prefill versus decode in LLM inference. Which phase is compute-bound and which is memory-bandwidth-bound, and why does that matter for serving?