Why it matters
Next-token prediction is the cheapest supervision available: any text is a labelled example, with the label being the text itself one position later. That removes annotation from pretraining entirely. The engineering problem is that the objective is local and unweighted — it treats every token as equally important and never directly optimises for the downstream task — so capability emerges from scale and data distribution rather than from the loss definition.
Mental model
Language modelling is an extremely well-read autocomplete: given the text so far, it assigns probabilities to what comes next. The metaphor breaks because autocomplete suggests a phrase while the model represents a full conditional distribution over the vocabulary, and the same distribution supports translation, arithmetic and code when the prefix frames it that way.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.
- positions
- 4
- candidates
- 4 ids
- cells
- 16
- row total
- 1.000
Step 1 of 4
One distribution per position
Every row is a full distribution over the same four candidate ids; the context changes the shape, not the column set. The tagged y cell is the observed next token, and the other three probabilities are part of the same normalised row.
Next: The target is one cell
Figure. Four positions in The cat sat on the mat, probabilities from illustrative logits. Rows are the context so far and columns the four candidate ids; a cell is the softmax probability the model assigns to that next token, and every row totals 1.000. The cell tagged y is the token that actually came next, so that row's loss is −ln of its printed probability. The column totals add the four printed rows: a marginal over four positions, not a corpus frequency.
What to notice. The target is one cell per row, but the other three cells are not ignored. Softmax divides by the sum of all four exponentials, which is why the printed probabilities, not the raw scores, enter the loss. The four row losses are 0.489, 0.501, 0.574 and 0.506 nats, so the mean is 0.518 and perplexity is e^0.518 = 1.68; both follow from the printed values. Against a uniform four-way baseline the same rows would cost ln 4 = 1.386 nats each, which is the reduction the objective produced. The logits are invented and four candidates hide the long tail a real vocabulary carries; the column totals therefore say nothing about how often those words occur in text.
Text version
A four by four matrix. Rows are the contexts The cat, cat sat, sat on and on the; columns are the candidate next tokens sat, on, the and mat. Each row is a probability distribution over the four columns and the row-total column reads 1.000. The top row prints 0.613, 0.167, 0.137 and 0.083, with sat tagged y. The second row prints 0.111, 0.606, 0.201 and 0.082, with on tagged y. The third prints 0.154, 0.207, 0.563 and 0.076, with its third cell tagged y. The fourth prints 0.082, 0.134, 0.181 and 0.603, with mat tagged y. Column totals are 0.960, 1.114, 1.082 and 0.844. Steps highlight the four target cells, the second row's target at 0.606 with loss 0.501, the four row losses with mean 0.518, and a uniform frame where every cell prints 0.250. The readout reports the context, the candidate and the printed probability with its row total.
- Tokenise a document, prepend the sequence-start token, and shift the labels one position left so each input predicts its successor.
- Run the model over the sequence and produce logits for every position in parallel.
- Apply a softmax and take the negative log probability of the observed next token at each position.
- Mask padding positions so they contribute nothing, and average over real tokens to get loss per token.
- Backpropagate through the sequence, which is why causal masking must prevent position t from seeing t+1.
- At inference, either sample from the distribution with a temperature and truncation policy, or search for a high-probability sequence.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Next-token objective | Unlimited unlabelled supervision, one objective for all tasks | No direct signal for a specific downstream metric | You need calibrated confidence on a narrow task |
| Masked-token objective | Bidirectional context, strong representations | Cannot generate left to right without adaptation | The product is text generation |
| Greedy decoding | Deterministic, cheap, reproducible | Repetitive and dull text, no diversity | Open-ended generation is the goal |
| Sampling with temperature | Diverse, natural text | Nondeterminism and possible incoherence | Outputs must be stable or auditable |
Good for
Unlimited unlabelled supervision, one objective for all tasks
Cost
No direct signal for a specific downstream metric
Fails when
You need calibrated confidence on a narrow task
- You need calibrated confidence on a narrow task
- The product is text generation
- Open-ended generation is the goal
- Outputs must be stable or auditable
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Next-token objective is good for Unlimited unlabelled supervision, one objective for all tasks, costs No direct signal for a specific downstream metric, and fails when You need calibrated confidence on a narrow task. Masked-token objective is good for Bidirectional context, strong representations, costs Cannot generate left to right without adaptation, and fails when The product is text generation. Greedy decoding is good for Deterministic, cheap, reproducible, costs Repetitive and dull text, no diversity, and fails when Open-ended generation is the goal. Sampling with temperature is good for Diverse, natural text, costs Nondeterminism and possible incoherence, and fails when Outputs must be stable or auditable
Failure modes
- Loss computed on shifted-by-zero labels. The model learns to copy its input and loss looks excellent while generation is broken. Debug: print one input id sequence and its label sequence and confirm the offset is exactly one.
- Perplexity compared across tokenizers. Two models look different in quality because their token inventories differ. Debug: compare bits per character instead, which normalises by the underlying text.
- Padding counted as targets. Loss includes pad positions and the model learns to emit padding. Debug: assert that the number of scored positions equals the number of non-pad tokens.
In production
Track loss per token and bits per character on a fixed held-out corpus, and watch for a rising share of truncation or padding in the batch. Decoding settings belong to the product config, so log them with each generation to keep evaluations comparable.
Common misconceptions
People think: Perplexity is a portable quality score.
Actually: It depends on the tokenizer, the vocabulary and the evaluation corpus, so it is only comparable between models sharing all three.
People think: Next-token prediction cannot learn anything beyond local statistics.
Actually: Predicting the next token well requires modelling long-range structure, facts and style, which is why the objective scales into general capability.
Interview angle
A strong answer writes the conditional likelihood, explains the shift-by-one label construction and the causal mask it requires, and distinguishes the learned distribution from the decoding policy. It should mention why perplexity is not portable. Weak answers call it 'predicting the next word' and cannot say what the loss is or why tokenizer choice affects the metric.
Sources
- Bengio et al., A Neural Probabilistic Language Model — the neural formulation of next-word prediction.
- Goodfellow, Bengio & Courville, Deep Learning — maximum likelihood, cross-entropy and perplexity.
- Holtzman et al., The Curious Case of Neural Text Degeneration — why decoding policy is separate from the learned distribution.