Skip to content
AI Engineer Guide.
seniorReportedcurated

Pre-norm versus post-norm: what changes, and why did essentially everyone move to pre-norm and RMSNorm?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why does warm-up interact with post-norm?
  • Post-norm puts a normalisation on the residual trunk, so the gradient each early layer receives has passed through every downstream norm.
  • Stacks in that arrangement need a warm-up ramp and can still diverge, which is why the placement is a training-stability choice rather than a cosmetic one.
  • Pre-norm keeps the identity path clean, so 80+ layer stacks train stably and tolerate a shorter, gentler warm-up than a post-norm stack of the same depth; warm-up is still part of the schedule.
Where else do norms appear in modern architectures?
  • Two per block, one before attention and one before the feed-forward, so the count scales with depth rather than with vocabulary or context length.
  • A final norm sits after the last block and before the LM head, which projects the last hidden state to vocabulary logits.
  • The gain vector is learned per feature, so switching between LayerNorm and RMSNorm changes the parameter set enough that a checkpoint cannot simply be converted.
  • Keep the reduction in float32 even when the rest of the block runs in lower precision, because squaring near-zero activations in half precision loses range.

Red flags

  • Claims pre-norm always achieves lower loss
  • Cannot say what RMSNorm omits relative to LayerNorm

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals