Why it matters
Next-token training runs all positions in parallel, which is only valid if position t cannot see t+1. Causal masking enforces that constraint inside the attention operation, so one forward pass yields a valid prediction at every position. It also keeps training and inference consistent: generation appends tokens to a prefix, and the mask guarantees the prefix's representations were computed the same way during training.
Mental model
Causal masking is a rule that each person in a queue may only see the people ahead of them, never behind. The metaphor breaks because the rule is enforced by adding negative infinity to scores, not by hiding information — the values are present in memory, just given zero weight after the softmax.
How it works
Hover or focus a cell to read its exact value and printed row total. Dashed cells are removed by the causal mask before normalisation.
Step 1 of 4
Mask on · row The
The has one allowed key, itself. The four dashed cells are future positions set to −∞ before the softmax; the surviving weight renormalises to 1.000.
Next: Row sat
Figure. One head over five tokens, with illustrative weights. Rows are queries, columns keys, and a cell is the weight that query places on that key. With the mask on, the upper triangle is removed before normalisation and each visible row is renormalised, so the printed row total stays 1.000.
What to notice. Masking happens before softmax. If the unmasked weights are wⱼ, the masked weight is wⱼ divided by the sum of the visible wⱼ, so renormalising visible cells is exactly the masked softmax, not an approximation. A hidden future cell carries exactly zero weight; that is why it is drawn empty rather than pale.
Text version
A five by five attention matrix for The, cat, sat, on and mat. Rows are queries and columns keys. The default frame has the causal mask on and the first query row, The, highlighted: only the diagonal cell is visible and prints 1.000, while the four future cells are dashed and empty. The row-total column reads 1.000 on every row. The next step focuses sat, where The, cat and sat print 0.195, 0.444 and 0.361 and the remaining two cells are masked. The next step turns the mask off to compare: the future cells fill with weights and every row still sums to 1.000. The last step turns the mask back on and focuses mat, whose row is fully visible and prints 0.080, 0.100, 0.120, 0.280 and 0.420. A checkbox toggles the mask at any step, and hovering a cell reports the query, key, printed weight and row total.
- Build a lower-triangular boolean matrix of shape sequence by sequence, true where attention is allowed.
- Convert it to an additive mask with zeros for allowed positions and a large negative constant for forbidden ones.
- Combine it with the padding mask by adding the two, so a position must pass both tests.
- Broadcast the mask over batch and head dimensions explicitly rather than relying on accidental alignment.
- Apply it to the scores before the softmax, then verify that each row's weights sum to one.
- Test the result by changing a future token and asserting earlier logits do not move.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Full causal mask | Exact training-generation parity | Quadratic score matrix and mask memory | Sequences are long |
| Sliding-window causal mask | Cheap long contexts, local coherence | Cannot read evidence beyond the window | Answers depend on distant text |
| Document-level masking | Prevents cross-document leakage in packed batches | More complex mask construction | Batches pack multiple short documents |
| Mask-free prefix training | Simpler kernels | Leaks future tokens and invalidates the objective | You care about generation quality |
Good for
Exact training-generation parity
Cost
Quadratic score matrix and mask memory
Fails when
Sequences are long
- Sequences are long
- Answers depend on distant text
- Batches pack multiple short documents
- You care about generation quality
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Full causal mask is good for Exact training-generation parity, costs Quadratic score matrix and mask memory, and fails when Sequences are long. Sliding-window causal mask is good for Cheap long contexts, local coherence, costs Cannot read evidence beyond the window, and fails when Answers depend on distant text. Document-level masking is good for Prevents cross-document leakage in packed batches, costs More complex mask construction, and fails when Batches pack multiple short documents. Mask-free prefix training is good for Simpler kernels, costs Leaks future tokens and invalidates the objective, and fails when You care about generation quality
Failure modes
- Mask shape broadcast to nothing. A mask of the wrong rank adds a scalar-like zero and the model reads the future. Debug: assert the mask shape equals the score shape after broadcasting, and run the future-token perturbation test.
- Mask applied after softmax. Forbidden positions keep probability mass because the normalisation already happened. Debug: inspect one attention row and check that future positions have exactly zero weight.
- Packed documents share a mask. Two short documents packed into one sequence attend to each other across the boundary. Debug: compare loss on packed versus unpacked batches of the same examples.
In production
Test the mask on every architecture change with a one-token perturbation, since a broken mask still trains and only shows up as suspiciously good loss. When packing sequences for throughput, build per-document masks and verify that boundary tokens have zero cross-document attention.
Common misconceptions
People think: Causal masking is only needed for generation.
Actually: It is needed during training precisely because training is parallel; without it the objective is invalid even though no generation happens.
People think: A causal mask also handles padding.
Actually: The two constraints are independent; padding positions can be in the past and still must be excluded, so the masks have to be combined.
Interview angle
A strong answer explains why parallel training needs the mask, how the additive form interacts with softmax, and how padding and document boundaries compose with it. It should offer the perturbation test as verification. Weak answers say 'it hides the future' without explaining why that matters for a training pass that never generates.
Sources
- Vaswani et al., Attention Is All You Need — masked decoder self-attention for autoregressive training.
- Radford et al., Improving Language Understanding by Generative Pre-Training — the causal objective and its training procedure.
- PyTorch, MultiheadAttention — how an attention mask is shaped and applied.