Why it matters
Labels are the expensive part of supervised learning, and raw text has none. A pretraining objective manufactures supervision from the data itself by hiding part of it and asking the model to recover it. The choice of what to hide determines the information the model must represent: hiding the future forces a generative model, hiding random tokens forces bidirectional understanding, and hiding a span forces reconstruction from surrounding context.
Mental model
The objective is the exam you set for a student with no teacher: cloze questions produce a reader who infers missing words, while predicting the next sentence produces a writer. The metaphor breaks because the same parameters can sit both exams in sequence, so the objective is a curriculum stage rather than a permanent aptitude.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.
- pairs
- 10
- mean per predicted target
- 10 ÷ 4 = 2.5
- position t₁
- given, 0 sources
- source t₅
- 0 targets
Step 1 of 4
Causal · past only
t₁ is given, never predicted, so its column has no sources. t₅ is the last token in the window, so it predicts nothing. The other targets draw on a growing past: 1, 2, 3 and 4 sources, which is the 10 pairs.
Next: Masked · all but itself
Figure. Five positions, illustrative pair counts. A cell is 1 when the target token in that column may be reconstructed from the source token in that row, and 0 when that source is not available for that target. Each objective draws a different pattern: causal prediction uses only earlier tokens, masked reconstruction uses every token except the target itself, and span corruption reconstructs a two-token gap from the three surrounding tokens. Row totals count the targets a source helps predict; column totals count the sources a target draws on.
What to notice. The causal grid is strictly upper-triangular with sources on rows and targets on columns, and it has 4 + 3 + 2 + 1 = 10 pairs, not 15. The attention mask on the causal-masking page includes the query's own position because a query may read itself; this grid cannot, because a token is the label it would predict. The masked grid has 5 × 4 = 20 pairs; the five diagonal pairs are the ones removed when the target's own token is hidden. The leak frame puts those five copy pairs back: every target can read itself, the model can copy, and loss falls while generation stays broken. Span corruption is 2 targets × 3 sources = 6 pairs. The counts are the objective's supervision graph, not a measurement of how much any model learned.
Text version
Four frames of a five by five source-target grid. Rows are sources t1 to t5 and columns targets t1 to t5; 1 means the target may be reconstructed from that source. The causal frame prints 1 only above the diagonal, with row totals 4, 3, 2, 1, 0 and column totals 0, 1, 2, 3, 4. The masked frame prints 0 on the diagonal and 1 elsewhere, with every row and column total 4. The span frame prints 1 only in columns t3 and t4 and only for sources t1, t2 and t5, with row totals 2, 2, 0, 0, 2 and column totals 0, 0, 3, 3, 0. The leak frame prints 1 everywhere, with every row and column total 5. Steps highlight the causal pair set, the masked diagonal, the span block and the leaked diagonal; the readout reports the source, the target and whether the pair is supervised.
- Choose what the model may see: only the past for causal prediction, the whole sequence for masked reconstruction, or two views for contrastive matching.
- Define the corruption: nothing for causal, random token replacement for masked language modelling, contiguous spans for denoising.
- Build targets from the uncorrupted sequence so the loss measures recovery rather than copying.
- Decide whether the loss applies to every position or only to corrupted ones, since that changes the effective learning signal per token.
- Train long enough on enough diverse text, because the objective is weak per example and capability comes from aggregate coverage.
- Match the downstream adaptation to the objective: a generative head for causal models, a classification head or span decoder for encoders.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Causal next token | Generation, in-context learning, one simple loss | Cannot see future context, slower per-token supervision | The task is classification over whole documents |
| Masked token prediction | Bidirectional representations, strong encoders | No native generation, train-test mismatch from mask tokens | The product must produce text |
| Span corruption | Good for seq2seq, efficient target length | Requires an encoder-decoder and span design | You need token-level likelihoods |
| Contrastive matching | Retrieval and embedding quality | Needs large batches or hard negatives | The task needs generation rather than similarity |
Good for
Generation, in-context learning, one simple loss
Cost
Cannot see future context, slower per-token supervision
Fails when
The task is classification over whole documents
- The task is classification over whole documents
- The product must produce text
- You need token-level likelihoods
- The task needs generation rather than similarity
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Causal next token is good for Generation, in-context learning, one simple loss, costs Cannot see future context, slower per-token supervision, and fails when The task is classification over whole documents. Masked token prediction is good for Bidirectional representations, strong encoders, costs No native generation, train-test mismatch from mask tokens, and fails when The product must produce text. Span corruption is good for Good for seq2seq, efficient target length, costs Requires an encoder-decoder and span design, and fails when You need token-level likelihoods. Contrastive matching is good for Retrieval and embedding quality, costs Needs large batches or hard negatives, and fails when The task needs generation rather than similarity
Failure modes
- Target leakage through the corruption. The original token remains visible somewhere in the input, so the model copies instead of inferring. Debug: decode the model input and confirm the target span is genuinely absent.
- Loss averaged over the wrong positions. Including uncorrupted tokens dilutes the signal and the model learns to copy. Debug: count scored positions against corrupted positions for one batch.
- Objective mismatched to adaptation. A masked encoder is fine-tuned as a generator and produces poor text. Debug: compare the adaptation head against the objective the checkpoint was trained with.
In production
Record the objective, corruption policy and masking ratio with the checkpoint, because fine-tuning behaviour depends on all three. When continuing pretraining on domain text, keep the original objective and mix in general data to avoid catastrophic drift.
Common misconceptions
People think: Masked language modelling and causal modelling learn the same representation.
Actually: They expose different context, so the encoder learns bidirectional associations while the causal model learns a conditional distribution; each transfers better to different tasks.
People think: The objective matters less than the data volume.
Actually: At equal data, the objective determines which capabilities are even representable, and a mismatch with the downstream task wastes most of the compute.
Interview angle
A strong answer maps each objective to the context it exposes and the capability it produces, then connects that to downstream adaptation. It should discuss target construction and which positions carry loss. Weak answers name BERT and GPT without explaining why masking or causality changes what the model can be used for.
Sources
- Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers — masked language modelling and next-sentence prediction.
- Radford et al., Improving Language Understanding by Generative Pre-Training — the causal pretraining objective and adaptation.
- Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — span corruption and a systematic comparison of objectives.