Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 9 of 38

Speculative decoding

0 of 38 complete

Why it matters

Decoding is sequential: each token needs a forward pass, and each pass uses far less compute than the device can deliver. Speculative decoding fills that idle capacity by guessing ahead with a cheaper model and checking the guesses in parallel with the target. The engineering problem is that guess quality is domain-specific and verification has a cost, so the technique helps only when acceptance is high and the batch is small enough to leave headroom.

Mental model

It is an assistant drafting sentences that a senior editor approves in one reading. The analogy breaks because the approval is probabilistic: a rejected word invalidates the rest of the draft from that point, and all work after the first disagreement is discarded.

How it works

Mechanism, step by stepInteractive
  1. 1Pick a draft source: a smaller model from the same family, a distilled head, or an n-gram lookup over the prompt.
  2. 2Choose the number of proposed tokens from measured acceptance and verification cost, not from the maximum the engine allows.
  3. 3Verify with the rejection rule that preserves the target distribution, and resample from the corrected distribution on rejection.
  4. 4Measure acceptance per workload slice, since code, structured output and free prose accept at very different rates.
  5. 5Compare end-to-end latency at the batch sizes you actually serve, because drafting competes with other requests for compute.
  6. 6Keep a non-speculative path available for the slices where acceptance is poor.

Step 1 of 6

Pick a draft source: a smaller model from the same family, a distilled head, or an n-gram lookup over the prompt.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Pick a draft source: a smaller model from the same family, a distilled head, or an n-gram… 2. Choose the number of proposed tokens from measured acceptance and verification cost, not … 3. Verify with the rejection rule that preserves the target distribution, and resample from … 4. Measure acceptance per workload slice, since code, structured output and free prose accep… 5. Compare end-to-end latency at the batch sizes you actually serve, because drafting compet… 6. Keep a non-speculative path available for the slices where acceptance is poor.

  1. Pick a draft source: a smaller model from the same family, a distilled head, or an n-gram lookup over the prompt.
  2. Choose the number of proposed tokens from measured acceptance and verification cost, not from the maximum the engine allows.
  3. Verify with the rejection rule that preserves the target distribution, and resample from the corrected distribution on rejection.
  4. Measure acceptance per workload slice, since code, structured output and free prose accept at very different rates.
  5. Compare end-to-end latency at the batch sizes you actually serve, because drafting competes with other requests for compute.
  6. Keep a non-speculative path available for the slices where acceptance is poor.

Trade-offs

ChoiceGood forCostFails when
Small draft modelGood acceptance on in-domain textExtra weights in memoryTraffic is out of the draft's domain
N-gram or prompt lookupNo extra model, strong on copy-heavy tasksWeak on novel phrasingOutput is creative rather than extracted
More proposed tokensFewer sequential target stepsWasted verification on rejected tailsAcceptance is low
Speculation at low batchUses idle computeNo benefit when the device is saturatedConcurrency is already high
Trade-off mapInteractive

Good for

Good acceptance on in-domain text

Cost

Extra weights in memory

Fails when

Traffic is out of the draft's domain

  1. Traffic is out of the draft's domain
  2. Output is creative rather than extracted
  3. Acceptance is low
  4. Concurrency is already high

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Small draft model is good for Good acceptance on in-domain text, costs Extra weights in memory, and fails when Traffic is out of the draft's domain. N-gram or prompt lookup is good for No extra model, strong on copy-heavy tasks, costs Weak on novel phrasing, and fails when Output is creative rather than extracted. More proposed tokens is good for Fewer sequential target steps, costs Wasted verification on rejected tails, and fails when Acceptance is low. Speculation at low batch is good for Uses idle compute, costs No benefit when the device is saturated, and fails when Concurrency is already high

Failure modes

  • Low acceptance on domain shift. Latency rises because every proposal is verified and rejected. Debug: log accepted tokens per proposal by request type and compare with the baseline path.
  • Distribution drift from wrong verification. Sampling without the correction step produces text that is subtly not from the target model. Debug: run a distribution test on a fixed prompt set against the non-speculative path.
  • Memory pressure from the draft. The extra weights reduce cache headroom and concurrency falls. Debug: compare maximum concurrent sequences with and without the draft loaded.

In production

Track acceptance rate, verified tokens per request and latency against a control path. Roll speculation out per workload slice, and disable it automatically when acceptance drops below the point where it pays.

Common misconceptions

People think: Speculative decoding makes the model smaller and therefore faster.

Actually: The target model still produces every accepted token; the gain comes from batching verification, not from reduced capability.

People think: It changes output quality.

Actually: With the correct rejection rule the output distribution is unchanged; only the number of sequential target passes changes.

Interview angle

Cover the accept-reject rule, why it preserves the distribution, and why acceptance decides the economics. Name the batch-size reversal. Describing it as running two models and taking the faster one is the answer to avoid.

Sources