seniorReportedcurated
Pre-norm versus post-norm: what changes, and why did essentially everyone move to pre-norm and RMSNorm?
30-second answer
- Post-norm computes Norm(x + Sublayer(x)), so every gradient on the residual trunk passes through a normalisation; pre-norm computes x + Sublayer(Norm(x)), leaving an unnormalised identity path.
- That clean identity path is why 80+ layer stacks train stably: the gradient reaching early layers is not attenuated by every downstream block.
- The trade-off is quality when both train, because post-norm can reach a slightly lower final loss than pre-norm when it does converge.
- RMSNorm drops mean-centring and the shift parameter, dividing by the root mean square and applying a learned gain, so it is cheaper per token and trains comparably.
- It is not invariant to adding a constant to every feature, so a layer whose outputs carry a large mean offset can saturate the next block.
Likely follow-ups
Open a question to rehearse the answer.
Why does warm-up interact with post-norm?
- Post-norm puts a normalisation on the residual trunk, so the gradient each early layer receives has passed through every downstream norm.
- Stacks in that arrangement need a warm-up ramp and can still diverge, which is why the placement is a training-stability choice rather than a cosmetic one.
- Pre-norm keeps the identity path clean, so 80+ layer stacks train stably and tolerate a shorter, gentler warm-up than a post-norm stack of the same depth; warm-up is still part of the schedule.
Where else do norms appear in modern architectures?
- Two per block, one before attention and one before the feed-forward, so the count scales with depth rather than with vocabulary or context length.
- A final norm sits after the last block and before the LM head, which projects the last hidden state to vocabulary logits.
- The gain vector is learned per feature, so switching between LayerNorm and RMSNorm changes the parameter set enough that a checkpoint cannot simply be converted.
- Keep the reduction in float32 even when the rest of the block runs in lower precision, because squaring near-zero activations in half precision loses range.
Red flags
- Claims pre-norm always achieves lower loss
- Cannot say what RMSNorm omits relative to LayerNorm
Concepts this tests
Tags
Source: github.com