Skip to content
AI Engineer Guide.
juniorReportedcurated

What is causal masking, why is it needed, and why apply it before the softmax?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why does a missing mask still produce a falling training loss?
  • With the mask absent or mis-shaped, position t can see token t+1, so the cheapest way to lower loss is to copy the answer instead of predicting it.
  • Loss falls faster than a correct model's would, which is the signature: the model is reading the target, not learning the distribution.
  • The check is a perturbation test: change a token at position t+1 and assert the logits at position t are unchanged.
  • Deployment exposes it immediately, because at inference there is no future token to copy.
How does FlashAttention exploit the causal structure?
  • It applies the mask inside the tile loop, so blocks lying entirely above the diagonal are skipped rather than computed and then discarded.
  • Tiles that straddle the diagonal still need the mask, and the online softmax keeps a running maximum and normaliser so partial results can be rescaled.
  • The saving is roughly half, because the allowed region is the lower triangle, and the result is unchanged because the kernel is exact rather than approximate.

Red flags

  • Masks after softmax and then forgets renormalization
  • Confuses the padding mask with the causal mask

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals