Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 13 of 33

Language modelling

0 of 33 complete

Why it matters

Next-token prediction is the cheapest supervision available: any text is a labelled example, with the label being the text itself one position later. That removes annotation from pretraining entirely. The engineering problem is that the objective is local and unweighted — it treats every token as equally important and never directly optimises for the downstream task — so capability emerges from scale and data distribution rather than from the loss definition.

Mental model

Language modelling is an extremely well-read autocomplete: given the text so far, it assigns probabilities to what comes next. The metaphor breaks because autocomplete suggests a phrase while the model represents a full conditional distribution over the vocabulary, and the same distribution supports translation, arithmetic and code when the prefix frames it that way.

How it works

Next token · four positions, one vocabularyInteractive

hover or focus a cell

01.000
context ↓candidate →
satonthematΣ
The cat1.000
cat sat1.000
sat on1.000
on the1.000
mass0.9601.1141.0820.844

Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.

positions
4
candidates
4 ids
cells
16
row total
1.000

Step 1 of 4

One distribution per position

Every row is a full distribution over the same four candidate ids; the context changes the shape, not the column set. The tagged y cell is the observed next token, and the other three probabilities are part of the same normalised row.

Next: The target is one cell

Step 1/4 · One distribution per position

Figure. Four positions in The cat sat on the mat, probabilities from illustrative logits. Rows are the context so far and columns the four candidate ids; a cell is the softmax probability the model assigns to that next token, and every row totals 1.000. The cell tagged y is the token that actually came next, so that row's loss is −ln of its printed probability. The column totals add the four printed rows: a marginal over four positions, not a corpus frequency.

What to notice. The target is one cell per row, but the other three cells are not ignored. Softmax divides by the sum of all four exponentials, which is why the printed probabilities, not the raw scores, enter the loss. The four row losses are 0.489, 0.501, 0.574 and 0.506 nats, so the mean is 0.518 and perplexity is e^0.518 = 1.68; both follow from the printed values. Against a uniform four-way baseline the same rows would cost ln 4 = 1.386 nats each, which is the reduction the objective produced. The logits are invented and four candidates hide the long tail a real vocabulary carries; the column totals therefore say nothing about how often those words occur in text.

Text version

A four by four matrix. Rows are the contexts The cat, cat sat, sat on and on the; columns are the candidate next tokens sat, on, the and mat. Each row is a probability distribution over the four columns and the row-total column reads 1.000. The top row prints 0.613, 0.167, 0.137 and 0.083, with sat tagged y. The second row prints 0.111, 0.606, 0.201 and 0.082, with on tagged y. The third prints 0.154, 0.207, 0.563 and 0.076, with its third cell tagged y. The fourth prints 0.082, 0.134, 0.181 and 0.603, with mat tagged y. Column totals are 0.960, 1.114, 1.082 and 0.844. Steps highlight the four target cells, the second row's target at 0.606 with loss 0.501, the four row losses with mean 0.518, and a uniform frame where every cell prints 0.250. The readout reports the context, the candidate and the printed probability with its row total.

  1. Tokenise a document, prepend the sequence-start token, and shift the labels one position left so each input predicts its successor.
  2. Run the model over the sequence and produce logits for every position in parallel.
  3. Apply a softmax and take the negative log probability of the observed next token at each position.
  4. Mask padding positions so they contribute nothing, and average over real tokens to get loss per token.
  5. Backpropagate through the sequence, which is why causal masking must prevent position t from seeing t+1.
  6. At inference, either sample from the distribution with a temperature and truncation policy, or search for a high-probability sequence.

Trade-offs

ChoiceGood forCostFails when
Next-token objectiveUnlimited unlabelled supervision, one objective for all tasksNo direct signal for a specific downstream metricYou need calibrated confidence on a narrow task
Masked-token objectiveBidirectional context, strong representationsCannot generate left to right without adaptationThe product is text generation
Greedy decodingDeterministic, cheap, reproducibleRepetitive and dull text, no diversityOpen-ended generation is the goal
Sampling with temperatureDiverse, natural textNondeterminism and possible incoherenceOutputs must be stable or auditable
Trade-off mapInteractive

Good for

Unlimited unlabelled supervision, one objective for all tasks

Cost

No direct signal for a specific downstream metric

Fails when

You need calibrated confidence on a narrow task

  1. You need calibrated confidence on a narrow task
  2. The product is text generation
  3. Open-ended generation is the goal
  4. Outputs must be stable or auditable

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Next-token objective is good for Unlimited unlabelled supervision, one objective for all tasks, costs No direct signal for a specific downstream metric, and fails when You need calibrated confidence on a narrow task. Masked-token objective is good for Bidirectional context, strong representations, costs Cannot generate left to right without adaptation, and fails when The product is text generation. Greedy decoding is good for Deterministic, cheap, reproducible, costs Repetitive and dull text, no diversity, and fails when Open-ended generation is the goal. Sampling with temperature is good for Diverse, natural text, costs Nondeterminism and possible incoherence, and fails when Outputs must be stable or auditable

Failure modes

  • Loss computed on shifted-by-zero labels. The model learns to copy its input and loss looks excellent while generation is broken. Debug: print one input id sequence and its label sequence and confirm the offset is exactly one.
  • Perplexity compared across tokenizers. Two models look different in quality because their token inventories differ. Debug: compare bits per character instead, which normalises by the underlying text.
  • Padding counted as targets. Loss includes pad positions and the model learns to emit padding. Debug: assert that the number of scored positions equals the number of non-pad tokens.

In production

Track loss per token and bits per character on a fixed held-out corpus, and watch for a rising share of truncation or padding in the batch. Decoding settings belong to the product config, so log them with each generation to keep evaluations comparable.

Common misconceptions

People think: Perplexity is a portable quality score.

Actually: It depends on the tokenizer, the vocabulary and the evaluation corpus, so it is only comparable between models sharing all three.

People think: Next-token prediction cannot learn anything beyond local statistics.

Actually: Predicting the next token well requires modelling long-range structure, facts and style, which is why the objective scales into general capability.

Interview angle

A strong answer writes the conditional likelihood, explains the shift-by-one label construction and the causal mask it requires, and distinguishes the learned distribution from the decoding policy. It should mention why perplexity is not portable. Weak answers call it 'predicting the next word' and cannot say what the loss is or why tokenizer choice affects the metric.

Sources