Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 25 of 33

Pretraining objectives

0 of 33 complete

Why it matters

Labels are the expensive part of supervised learning, and raw text has none. A pretraining objective manufactures supervision from the data itself by hiding part of it and asking the model to recover it. The choice of what to hide determines the information the model must represent: hiding the future forces a generative model, hiding random tokens forces bidirectional understanding, and hiding a span forces reconstruction from surrounding context.

Mental model

The objective is the exam you set for a student with no teacher: cloze questions produce a reader who infers missing words, while predicting the next sentence produces a writer. The metaphor breaks because the same parameters can sit both exams in sequence, so the objective is a curriculum stage rather than a permanent aptitude.

How it works

Pretraining · which source reconstructs which targetInteractive

hover or focus a cell

01
source ↓target →
t₁t₂t₃t₄t₅targets
t₁4
t₂3
t₃2
t₄1
t₅0
sources01234

Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.

pairs
10
mean per predicted target
10 ÷ 4 = 2.5
position t₁
given, 0 sources
source t₅
0 targets

Step 1 of 4

Causal · past only

t₁ is given, never predicted, so its column has no sources. t₅ is the last token in the window, so it predicts nothing. The other targets draw on a growing past: 1, 2, 3 and 4 sources, which is the 10 pairs.

Next: Masked · all but itself

Step 1/4 · Causal · past only

Figure. Five positions, illustrative pair counts. A cell is 1 when the target token in that column may be reconstructed from the source token in that row, and 0 when that source is not available for that target. Each objective draws a different pattern: causal prediction uses only earlier tokens, masked reconstruction uses every token except the target itself, and span corruption reconstructs a two-token gap from the three surrounding tokens. Row totals count the targets a source helps predict; column totals count the sources a target draws on.

What to notice. The causal grid is strictly upper-triangular with sources on rows and targets on columns, and it has 4 + 3 + 2 + 1 = 10 pairs, not 15. The attention mask on the causal-masking page includes the query's own position because a query may read itself; this grid cannot, because a token is the label it would predict. The masked grid has 5 × 4 = 20 pairs; the five diagonal pairs are the ones removed when the target's own token is hidden. The leak frame puts those five copy pairs back: every target can read itself, the model can copy, and loss falls while generation stays broken. Span corruption is 2 targets × 3 sources = 6 pairs. The counts are the objective's supervision graph, not a measurement of how much any model learned.

Text version

Four frames of a five by five source-target grid. Rows are sources t1 to t5 and columns targets t1 to t5; 1 means the target may be reconstructed from that source. The causal frame prints 1 only above the diagonal, with row totals 4, 3, 2, 1, 0 and column totals 0, 1, 2, 3, 4. The masked frame prints 0 on the diagonal and 1 elsewhere, with every row and column total 4. The span frame prints 1 only in columns t3 and t4 and only for sources t1, t2 and t5, with row totals 2, 2, 0, 0, 2 and column totals 0, 0, 3, 3, 0. The leak frame prints 1 everywhere, with every row and column total 5. Steps highlight the causal pair set, the masked diagonal, the span block and the leaked diagonal; the readout reports the source, the target and whether the pair is supervised.

  1. Choose what the model may see: only the past for causal prediction, the whole sequence for masked reconstruction, or two views for contrastive matching.
  2. Define the corruption: nothing for causal, random token replacement for masked language modelling, contiguous spans for denoising.
  3. Build targets from the uncorrupted sequence so the loss measures recovery rather than copying.
  4. Decide whether the loss applies to every position or only to corrupted ones, since that changes the effective learning signal per token.
  5. Train long enough on enough diverse text, because the objective is weak per example and capability comes from aggregate coverage.
  6. Match the downstream adaptation to the objective: a generative head for causal models, a classification head or span decoder for encoders.

Trade-offs

ChoiceGood forCostFails when
Causal next tokenGeneration, in-context learning, one simple lossCannot see future context, slower per-token supervisionThe task is classification over whole documents
Masked token predictionBidirectional representations, strong encodersNo native generation, train-test mismatch from mask tokensThe product must produce text
Span corruptionGood for seq2seq, efficient target lengthRequires an encoder-decoder and span designYou need token-level likelihoods
Contrastive matchingRetrieval and embedding qualityNeeds large batches or hard negativesThe task needs generation rather than similarity
Trade-off mapInteractive

Good for

Generation, in-context learning, one simple loss

Cost

Cannot see future context, slower per-token supervision

Fails when

The task is classification over whole documents

  1. The task is classification over whole documents
  2. The product must produce text
  3. You need token-level likelihoods
  4. The task needs generation rather than similarity

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Causal next token is good for Generation, in-context learning, one simple loss, costs Cannot see future context, slower per-token supervision, and fails when The task is classification over whole documents. Masked token prediction is good for Bidirectional representations, strong encoders, costs No native generation, train-test mismatch from mask tokens, and fails when The product must produce text. Span corruption is good for Good for seq2seq, efficient target length, costs Requires an encoder-decoder and span design, and fails when You need token-level likelihoods. Contrastive matching is good for Retrieval and embedding quality, costs Needs large batches or hard negatives, and fails when The task needs generation rather than similarity

Failure modes

  • Target leakage through the corruption. The original token remains visible somewhere in the input, so the model copies instead of inferring. Debug: decode the model input and confirm the target span is genuinely absent.
  • Loss averaged over the wrong positions. Including uncorrupted tokens dilutes the signal and the model learns to copy. Debug: count scored positions against corrupted positions for one batch.
  • Objective mismatched to adaptation. A masked encoder is fine-tuned as a generator and produces poor text. Debug: compare the adaptation head against the objective the checkpoint was trained with.

In production

Record the objective, corruption policy and masking ratio with the checkpoint, because fine-tuning behaviour depends on all three. When continuing pretraining on domain text, keep the original objective and mix in general data to avoid catastrophic drift.

Common misconceptions

People think: Masked language modelling and causal modelling learn the same representation.

Actually: They expose different context, so the encoder learns bidirectional associations while the causal model learns a conditional distribution; each transfers better to different tasks.

People think: The objective matters less than the data volume.

Actually: At equal data, the objective determines which capabilities are even representable, and a mismatch with the downstream task wastes most of the compute.

Interview angle

A strong answer maps each objective to the context it exposes and the capability it produces, then connects that to downstream adaptation. It should discuss target construction and which positions carry loss. Weak answers name BERT and GPT without explaining why masking or causality changes what the model can be used for.

Sources