Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 21 of 33

Causal masking

0 of 33 complete

Why it matters

Next-token training runs all positions in parallel, which is only valid if position t cannot see t+1. Causal masking enforces that constraint inside the attention operation, so one forward pass yields a valid prediction at every position. It also keeps training and inference consistent: generation appends tokens to a prefix, and the mask guarantees the prefix's representations were computed the same way during training.

Mental model

Causal masking is a rule that each person in a queue may only see the people ahead of them, never behind. The metaphor breaks because the rule is enforced by adding negative infinity to scores, not by hiding information — the values are present in memory, just given zero weight after the softmax.

How it works

Causal mask · one head, five tokensInteractive
01masked
query ↓key →
ThecatsatonmatΣ
The1.000
cat1.000
sat1.000
on1.000
mat1.000

Hover or focus a cell to read its exact value and printed row total. Dashed cells are removed by the causal mask before normalisation.

Step 1 of 4

Mask on · row The

The has one allowed key, itself. The four dashed cells are future positions set to −∞ before the softmax; the surviving weight renormalises to 1.000.

Next: Row sat

Step 1/4 · Mask on · row The

Figure. One head over five tokens, with illustrative weights. Rows are queries, columns keys, and a cell is the weight that query places on that key. With the mask on, the upper triangle is removed before normalisation and each visible row is renormalised, so the printed row total stays 1.000.

What to notice. Masking happens before softmax. If the unmasked weights are wⱼ, the masked weight is wⱼ divided by the sum of the visible wⱼ, so renormalising visible cells is exactly the masked softmax, not an approximation. A hidden future cell carries exactly zero weight; that is why it is drawn empty rather than pale.

Text version

A five by five attention matrix for The, cat, sat, on and mat. Rows are queries and columns keys. The default frame has the causal mask on and the first query row, The, highlighted: only the diagonal cell is visible and prints 1.000, while the four future cells are dashed and empty. The row-total column reads 1.000 on every row. The next step focuses sat, where The, cat and sat print 0.195, 0.444 and 0.361 and the remaining two cells are masked. The next step turns the mask off to compare: the future cells fill with weights and every row still sums to 1.000. The last step turns the mask back on and focuses mat, whose row is fully visible and prints 0.080, 0.100, 0.120, 0.280 and 0.420. A checkbox toggles the mask at any step, and hovering a cell reports the query, key, printed weight and row total.

  1. Build a lower-triangular boolean matrix of shape sequence by sequence, true where attention is allowed.
  2. Convert it to an additive mask with zeros for allowed positions and a large negative constant for forbidden ones.
  3. Combine it with the padding mask by adding the two, so a position must pass both tests.
  4. Broadcast the mask over batch and head dimensions explicitly rather than relying on accidental alignment.
  5. Apply it to the scores before the softmax, then verify that each row's weights sum to one.
  6. Test the result by changing a future token and asserting earlier logits do not move.

Trade-offs

ChoiceGood forCostFails when
Full causal maskExact training-generation parityQuadratic score matrix and mask memorySequences are long
Sliding-window causal maskCheap long contexts, local coherenceCannot read evidence beyond the windowAnswers depend on distant text
Document-level maskingPrevents cross-document leakage in packed batchesMore complex mask constructionBatches pack multiple short documents
Mask-free prefix trainingSimpler kernelsLeaks future tokens and invalidates the objectiveYou care about generation quality
Trade-off mapInteractive

Good for

Exact training-generation parity

Cost

Quadratic score matrix and mask memory

Fails when

Sequences are long

  1. Sequences are long
  2. Answers depend on distant text
  3. Batches pack multiple short documents
  4. You care about generation quality

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full causal mask is good for Exact training-generation parity, costs Quadratic score matrix and mask memory, and fails when Sequences are long. Sliding-window causal mask is good for Cheap long contexts, local coherence, costs Cannot read evidence beyond the window, and fails when Answers depend on distant text. Document-level masking is good for Prevents cross-document leakage in packed batches, costs More complex mask construction, and fails when Batches pack multiple short documents. Mask-free prefix training is good for Simpler kernels, costs Leaks future tokens and invalidates the objective, and fails when You care about generation quality

Failure modes

  • Mask shape broadcast to nothing. A mask of the wrong rank adds a scalar-like zero and the model reads the future. Debug: assert the mask shape equals the score shape after broadcasting, and run the future-token perturbation test.
  • Mask applied after softmax. Forbidden positions keep probability mass because the normalisation already happened. Debug: inspect one attention row and check that future positions have exactly zero weight.
  • Packed documents share a mask. Two short documents packed into one sequence attend to each other across the boundary. Debug: compare loss on packed versus unpacked batches of the same examples.

In production

Test the mask on every architecture change with a one-token perturbation, since a broken mask still trains and only shows up as suspiciously good loss. When packing sequences for throughput, build per-document masks and verify that boundary tokens have zero cross-document attention.

Common misconceptions

People think: Causal masking is only needed for generation.

Actually: It is needed during training precisely because training is parallel; without it the objective is invalid even though no generation happens.

People think: A causal mask also handles padding.

Actually: The two constraints are independent; padding positions can be in the past and still must be excluded, so the masks have to be combined.

Interview angle

A strong answer explains why parallel training needs the mask, how the additive form interacts with softmax, and how padding and document boundaries compose with it. It should offer the perturbation test as verification. Weak answers say 'it hides the future' without explaining why that matters for a training pass that never generates.

Sources