Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 13 of 28

Decoding

0 of 28 complete

Why it matters

Two models with identical weights can produce very different products depending on how you decode. A support classifier needs the same answer for the same ticket. A brainstorming tool needs plausible variety. A code generator needs syntactic validity, which may mean constraining the token choices rather than trusting the distribution.

Decoding is also where a lot of "the model is bad" reports actually originate. Repetition loops, rambling answers, empty responses and invalid JSON are frequently decoding or truncation artefacts. Understanding the selection rule lets you diagnose them without changing the model.

Mental model

Walking down a probability tree. At each step the model offers a distribution over branches; the decoder chooses one, moves to that node, and asks again. Greedy takes the widest branch every time. Beam search keeps several paths alive and picks the best at the end. Sampling draws a branch at random, weighted by probability.

The analogy breaks because the tree is astronomically wide and the decoder is myopic. It cannot know that a locally likely branch leads to a dead end three sentences later. Every practical method is a heuristic over an intractable search, which is why no decoder is "correct" in general.

How it works

  1. 1Take the logits for the next token from the final hidden state.
  2. 2Apply temperature to reshape the distribution, if sampling.
  3. 3Truncate low-probability candidates with top-k, top-p or min-p.
  4. 4Mask tokens that would violate a grammar or schema, if constrained decoding is on.
  5. 5Renormalise over what remains and select a token — argmax, beam expansion, or a draw.
  6. 6Append the token, check the stop conditions, and repeat.

Greedy decoding takes the highest-probability token at every step. It is fast, deterministic in exact arithmetic, and prone to repetition: once a phrase is likely, repeating it remains likely. Beam search keeps kk partial sequences and expands each one, scoring complete sequences at the end. With a length penalty it often finds a higher-likelihood sequence than greedy, but higher likelihood is not the same as better text. Likelihood-trained models assign high probability to bland, repetitive continuations, which is the core observation behind nucleus sampling.

Sampling draws from the distribution. Temperature flattens or sharpens it. Truncation removes the tail so that the draw cannot land on a very unlikely token. The three common truncation rules differ: top-k keeps a fixed number of candidates; top-p keeps the smallest set whose cumulative probability exceeds a threshold; min-p keeps tokens above a fraction of the top token's probability, so the candidate set adapts to how confident the model is.

Constrained decoding intersects the distribution with a formal language before selection. A grammar or JSON schema is compiled into a state machine, and any token that would make the prefix invalid is masked out. The output is guaranteed to parse, not guaranteed to be true.

Speculative decoding is orthogonal to selection: a small draft model proposes several tokens and the target model verifies them in one pass. It changes speed, not the distribution, when implemented with the correct acceptance rule.

The math of selection

Temperature scaling divides logits before the softmax:

pi(T)=exp⁡(zi/T)∑jexp⁡(zj/T)p_i(T) = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

As T→0T \to 0 the distribution collapses toward the argmax; as TT rises it flattens. Top-p keeps the smallest set SS such that:

∑i∈Spi≥p\sum_{i \in S} p_i \ge p

and renormalises within SS. Beam search maximises a length-normalised log-probability:

score(y)=1∣y∣α∑t=1∣y∣log⁡pθ(yt∣y<t,x)\text{score}(y) = \frac{1}{|y|^{\alpha}} \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x)

The exponent α\alpha controls the length bias. Without it, beam search favours shorter sequences because each additional term is a negative log-probability.

A sampler in twenty lines

import numpy as np
 
def sample_next(logits, temperature=1.0, top_p=0.9, rng=None):
    rng = rng or np.random.default_rng()
    logits = np.asarray(logits, dtype=np.float64)
 
    if temperature <= 0:
        return int(np.argmax(logits))
 
    probs = np.exp((logits - logits.max()) / temperature)
    probs /= probs.sum()
 
    order = np.argsort(probs)[::-1]
    cumulative = np.cumsum(probs[order])
    cutoff = int(np.searchsorted(cumulative, top_p)) + 1
    keep = order[:cutoff]
 
    truncated = np.zeros_like(probs)
    truncated[keep] = probs[keep]
    truncated /= truncated.sum()
    return int(rng.choice(len(truncated), p=truncated))

This is the mechanism, not a serving implementation. Production decoders handle batching, per-sequence generators, repetition penalties and grammar masks, and they must do so without breaking the statistical guarantee of the truncation rule.

Trade-offs

DecoderGood forCostFails when
GreedyExtraction, classification, deterministic formatsRepetition and bland continuationsThe task needs exploration or has ties
Beam searchTranslation, short constrained generationk times the compute, length biasOpen-ended text or long outputs
Top-p samplingChat, drafting, brainstormingRun-to-run varianceThe product needs a stable answer
Constrained decodingJSON, SQL, DSLs, tool argumentsGrammar compilation, possible quality lossThe constraint is wrong or too tight
Sample and voteReasoning tasks with checkable answersSeveral full generationsAnswers cannot be compared automatically
Trade-off mapInteractive

Good for

Extraction, classification, deterministic formats

Cost

Repetition and bland continuations

Fails when

The task needs exploration or has ties

  1. The task needs exploration or has ties
  2. Open-ended text or long outputs
  3. The product needs a stable answer
  4. The constraint is wrong or too tight
  5. Answers cannot be compared automatically

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Greedy is good for Extraction, classification, deterministic formats, costs Repetition and bland continuations, and fails when The task needs exploration or has ties. Beam search is good for Translation, short constrained generation, costs k times the compute, length bias, and fails when Open-ended text or long outputs. Top-p sampling is good for Chat, drafting, brainstorming, costs Run-to-run variance, and fails when The product needs a stable answer. Constrained decoding is good for JSON, SQL, DSLs, tool arguments, costs Grammar compilation, possible quality loss, and fails when The constraint is wrong or too tight. Sample and vote is good for Reasoning tasks with checkable answers, costs Several full generations, and fails when Answers cannot be compared automatically

Failure modes

Repetition loops. Greedy or low-temperature decoding settles into a cycle. Debug by inspecting the top-two logit margin across steps; if the margin is small, the model is uncertain and argmax is arbitrary. Raise temperature slightly or add a repetition penalty, then check that the fix does not break format compliance.

Rambling with high temperature. The tail gets sampled and the model drifts. Debug by lowering top-p before lowering temperature; truncation removes incoherent candidates, while temperature alone reshapes the whole distribution.

Valid but useless constrained output. Grammar masking forces a parseable structure, so the model fills required fields with plausible nonsense. Debug by comparing schema-valid samples against a semantic check; the constraint was never the problem.

Stop sequence appears in content. A stop string is matched inside legitimate text and generation ends early. Debug by logging the finish reason and the last tokens; choose stop sequences that cannot occur in the payload, or use a structured end token.

Failure modes, one at a timeInteractive

Symptom

Greedy or low-temperature decoding settles into a cycle.

Debug step

Inspecting the top-two logit margin across steps; if the margin is small, the model is uncertain and argmax is arbitrary. Raise temperature slightly or add a repetition penalty, then check that the fix does not break format compliance.

Figure. Each failure with its symptom and the check that would confirm it.

What to notice. A failure without a debug step is a rumour. Pick the symptom you are seeing; the check is the next action.

Text version

Failure modes: Repetition loops: Greedy or low-temperature decoding settles into a cycle. Debug by inspecting the top-two logit margin across steps; if the margin is small, the model is uncertain and argmax is arbitrary. Raise temperature slightly or add a repetition penalty, then check that the fix does not break format compliance. | Rambling with high temperature: The tail gets sampled and the model drifts. Debug by lowering top-p before lowering temperature; truncation removes incoherent candidates, while temperature alone reshapes the whole distribution. | Valid but useless constrained output: Grammar masking forces a parseable structure, so the model fills required fields with plausible nonsense. Debug by comparing schema-valid samples against a semantic check; the constraint was never the problem. | Stop sequence appears in content: A stop string is matched inside legitimate text and generation ends early. Debug by logging the finish reason and the last tokens; choose stop sequences that cannot occur in the payload, or use a structured end token.

In production

Choose the decoder per endpoint and record it with the prompt version. Log the parameters, finish reason and output length for every call, because a change in any of them is a behaviour change. For sampling endpoints, evaluate with several samples and report pass@k rather than a single run. Cap output length so that a decoding loop cannot become a cost incident, and keep a deterministic path for extraction tasks where reproducibility matters.

Common misconceptions

People think: Beam search is always better because it searches more.

Actually: Beam search optimises likelihood, and likelihood-trained models prefer bland, repetitive text. For open-ended generation, sampling usually reads better.

People think: Greedy decoding is fully deterministic in production.

Actually: Argmax is deterministic in exact arithmetic, but floating-point reduction order and batching can change logits slightly. Ties can flip the output.

Interview angle

Decoding questions reveal whether you can connect a modelling detail to product behaviour. Expect "why does the bot repeat itself" or "how would you guarantee valid JSON". The strong answer names the selection rule, the truncation strategy, the failure each one causes, and the evaluation that catches it. Practise with LLM fundamentals and prompting.

Sources