Skip to content
AI Engineer Guide.
seniorReportedcurated

Explain speculative decoding. Why is it lossless, and when does it stop helping?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why does the benefit shrink at high batch size?
  • Speculation spends idle compute on drafting, and at high batch there is no idle compute left because the device is already saturated with real requests.
  • The draft's proposals also compete for the same memory and bandwidth, so the cost shows up as lower throughput rather than lower latency.
  • Acceptance rate alone does not predict the win, because the comparison has to be end-to-end latency at the batch sizes you actually serve.
  • That is why it is rolled out per workload slice against a control path and disabled when it stops paying.
How would you measure the gain in a live deployment?
  • Track acceptance rate and verified tokens per request by request type, because code, structured output and prose accept at very different rates.
  • Compare end-to-end latency against a non-speculative control path at the same batch size rather than against a historical baseline.
  • Watch maximum concurrent sequences with and without the draft loaded, since extra weights reduce cache headroom.
  • Add a distribution test on a fixed prompt set to catch a broken verification rule that changes outputs subtly.

Red flags

  • Calls it an approximation that trades quality for speed
  • Ignores the high-batch regime where it hurts

Concepts this tests

Tags

Source: github.com

More in LLM & transformer fundamentals