juniorReportedcurated
What is causal masking, why is it needed, and why apply it before the softmax?
30-second answer
- The causal mask sets scores for future positions to negative infinity before the softmax, so position i can only attend to j ≤ i.
- That matches autoregressive inference, and it lets one forward pass over n tokens produce n valid next-token predictions in parallel instead of n sequential passes.
- Applying it before the softmax means the surviving weights renormalise automatically; masking afterwards leaves probability mass on forbidden positions.
- Without it the model can read the token it is meant to predict, training loss looks excellent, and the model learns nothing useful.
- Encoders omit the mask to get bidirectional context, and decoder batching still needs a separate padding mask so pad tokens are never attended.
- The cost is that a decoder-only model cannot revise an earlier token, so an early mistake propagates through everything after it.
Likely follow-ups
Open a question to rehearse the answer.
Why does a missing mask still produce a falling training loss?
- With the mask absent or mis-shaped, position t can see token t+1, so the cheapest way to lower loss is to copy the answer instead of predicting it.
- Loss falls faster than a correct model's would, which is the signature: the model is reading the target, not learning the distribution.
- The check is a perturbation test: change a token at position t+1 and assert the logits at position t are unchanged.
- Deployment exposes it immediately, because at inference there is no future token to copy.
How does FlashAttention exploit the causal structure?
- It applies the mask inside the tile loop, so blocks lying entirely above the diagonal are skipped rather than computed and then discarded.
- Tiles that straddle the diagonal still need the mask, and the online softmax keeps a running maximum and normaliser so partial results can be rescaled.
- The saving is roughly half, because the allowed region is the lower triangle, and the result is unchanged because the kernel is exact rather than approximate.
Red flags
- Masks after softmax and then forgets renormalization
- Confuses the padding mask with the causal mask
Concepts this tests
Tags
Source: github.com