Why it matters
A transformer block is a long chain of matrix multiplications, and gradients through a chain shrink or explode with depth. The residual connection turns the chain into a sum of increments, so the identity path carries gradient regardless of how many blocks sit between. Normalisation then keeps the stream's scale stable as those increments accumulate, which is what allows dozens of layers to train with a single learning rate.
Mental model
The residual stream is a shared document that each block edits by appending a note, rather than rewriting the page. Normalisation is the style guide that keeps every page at a readable size. The metaphor breaks because the notes are added numerically, not appended, so a later block can cancel an earlier one entirely.
How it works
Pre-norm · Step 1 of 4
Copy the stream
The block reads the residual stream and makes a normalised copy for the branch. Nothing has been added yet.
Active layers
- Residual stream in
Hover a layer to read what it holds.
Next: Normalise the branch
- Normalisation
- inside the branch
- Residual path
- never rescaled
- Warmup
- not required for depth
Figure. Two separate blocks, one per arrangement, selected with the switch. In pre-norm, LayerNorm sits inside the branch and the residual stream leaves unscaled; in post-norm, the sum is normalised on the main path. Each has its own input and output, and the feed-forward sublayer is omitted from both.
What to notice. The two arrangements are alternatives, not stages: a model uses one throughout. The add is identical; only the normalisation moves. Pre-norm leaves an unscaled identity path from input to output, which is why it tolerates depth without a long warmup. Post-norm rescales the whole stream, so early training needs warmup and careful initialisation.
Text version
A switch labelled Arrangement selects one of two complete stacks. Pre-norm: the stream enters, LayerNorm normalises the branch, Attention runs on that copy, and its output is added to an unscaled identity edge from the input; the stream leaves as an incremental update. Post-norm: Attention runs on the raw stream, its output is added to the identity edge, and LayerNorm then normalises the sum before the stream leaves. Each stack has its own input and output and four steps; no edge joins the two arrangements.
- Compute the sublayer output on a normalised copy of the stream, leaving the original untouched.
- Add that output back to the unnormalised stream, so the block contributes a delta rather than a replacement.
- For LayerNorm, subtract the mean across features and divide by the standard deviation, then apply a learned scale and shift.
- In the pre-norm arrangement, place the normalisation before attention and before the feed-forward block, and normalise once more before the output head.
- In the post-norm arrangement, add first and normalise after, accepting that early training needs warmup.
- Check gradient norms at the first and last block; the residual path should keep them within the same order of magnitude.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Pre-norm | Stable deep training, no warmup requirement | Slightly weaker final quality at matched compute in some reports | You need the last increment of quality and can afford tuning |
| Post-norm | Strong final quality, original transformer form | Warmup and initialisation sensitivity | Depth is large and the schedule is short |
| LayerNorm | Batch-independent, works with any sequence length | Two reductions per token and a learned shift | Kernel fusion or quantisation is the priority |
| No normalisation | Fewer operations and parameters | Activation scale drifts with depth | The model has more than a few layers |
Good for
Stable deep training, no warmup requirement
Cost
Slightly weaker final quality at matched compute in some reports
Fails when
You need the last increment of quality and can afford tuning
- You need the last increment of quality and can afford tuning
- Depth is large and the schedule is short
- Kernel fusion or quantisation is the priority
- The model has more than a few layers
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Pre-norm is good for Stable deep training, no warmup requirement, costs Slightly weaker final quality at matched compute in some reports, and fails when You need the last increment of quality and can afford tuning. Post-norm is good for Strong final quality, original transformer form, costs Warmup and initialisation sensitivity, and fails when Depth is large and the schedule is short. LayerNorm is good for Batch-independent, works with any sequence length, costs Two reductions per token and a learned shift, and fails when Kernel fusion or quantisation is the priority. No normalisation is good for Fewer operations and parameters, costs Activation scale drifts with depth, and fails when The model has more than a few layers
Failure modes
- Residual added after the normalisation. The stream is repeatedly rescaled and the identity path disappears, so deep stacks train slowly. Debug: trace the block's input and output and confirm the add uses the pre-normalisation tensor.
- Normalising over the wrong axis. Statistics are computed across the batch or sequence instead of features, coupling examples and positions. Debug: check that the normalised tensor has zero mean along the feature axis for a single token.
- Epsilon too small for the dtype. Variance near zero produces huge values or infinities in half precision. Debug: search for non-finite activations after each block on a batch with near-constant inputs.
In production
Log the norm of the residual stream at a few depths on a fixed batch; a steady climb across layers is an early sign of instability. When quantising, keep the normalisation in higher precision, because its reduction is the most sensitive operation in the block.
Common misconceptions
People think: Residual connections mainly help by adding capacity.
Actually: They add no parameters; their role is to provide a gradient path and let blocks learn increments, which is why very deep models need them.
People think: LayerNorm and batch norm are interchangeable normalisations.
Actually: Batch norm uses statistics across examples, so it couples the batch and behaves differently at inference; LayerNorm is per token and batch-independent.
Interview angle
A strong answer explains the identity gradient path, distinguishes pre- and post-norm on where normalisation sits, and says what that does to optimisation. It should note that LayerNorm is batch-independent. Weak answers say residuals 'help gradients flow' without being able to say why normalisation placement changes warmup needs.
Sources
- He et al., Deep Residual Learning for Image Recognition — the residual formulation and its optimisation effect.
- Ba, Kiros & Hinton, Layer Normalization — per-token normalisation independent of batch size.
- Xiong et al., On Layer Normalization in the Transformer Architecture — why pre-norm avoids warmup and how post-norm differs.