Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 24 of 33

Residuals & layer norm

0 of 33 complete

Why it matters

A transformer block is a long chain of matrix multiplications, and gradients through a chain shrink or explode with depth. The residual connection turns the chain into a sum of increments, so the identity path carries gradient regardless of how many blocks sit between. Normalisation then keeps the stream's scale stable as those increments accumulate, which is what allows dozens of layers to train with a single learning rate.

Mental model

The residual stream is a shared document that each block edits by appending a note, rather than rewriting the page. Normalisation is the style guide that keeps every page at a readable size. The metaphor breaks because the notes are added numerically, not appended, so a later block can cancel an earlier one entirely.

How it works

Where the normalisation sits: before the branch or after the addInteractive
Arrangement
signalresidual

Pre-norm · Step 1 of 4

Copy the stream

The block reads the residual stream and makes a normalised copy for the branch. Nothing has been added yet.

Active layers

  • Residual stream in

Hover a layer to read what it holds.

Next: Normalise the branch

Normalisation
inside the branch
Residual path
never rescaled
Warmup
not required for depth
Step 1/4 · Copy the stream

Figure. Two separate blocks, one per arrangement, selected with the switch. In pre-norm, LayerNorm sits inside the branch and the residual stream leaves unscaled; in post-norm, the sum is normalised on the main path. Each has its own input and output, and the feed-forward sublayer is omitted from both.

What to notice. The two arrangements are alternatives, not stages: a model uses one throughout. The add is identical; only the normalisation moves. Pre-norm leaves an unscaled identity path from input to output, which is why it tolerates depth without a long warmup. Post-norm rescales the whole stream, so early training needs warmup and careful initialisation.

Text version

A switch labelled Arrangement selects one of two complete stacks. Pre-norm: the stream enters, LayerNorm normalises the branch, Attention runs on that copy, and its output is added to an unscaled identity edge from the input; the stream leaves as an incremental update. Post-norm: Attention runs on the raw stream, its output is added to the identity edge, and LayerNorm then normalises the sum before the stream leaves. Each stack has its own input and output and four steps; no edge joins the two arrangements.

  1. Compute the sublayer output on a normalised copy of the stream, leaving the original untouched.
  2. Add that output back to the unnormalised stream, so the block contributes a delta rather than a replacement.
  3. For LayerNorm, subtract the mean across features and divide by the standard deviation, then apply a learned scale and shift.
  4. In the pre-norm arrangement, place the normalisation before attention and before the feed-forward block, and normalise once more before the output head.
  5. In the post-norm arrangement, add first and normalise after, accepting that early training needs warmup.
  6. Check gradient norms at the first and last block; the residual path should keep them within the same order of magnitude.

Trade-offs

ChoiceGood forCostFails when
Pre-normStable deep training, no warmup requirementSlightly weaker final quality at matched compute in some reportsYou need the last increment of quality and can afford tuning
Post-normStrong final quality, original transformer formWarmup and initialisation sensitivityDepth is large and the schedule is short
LayerNormBatch-independent, works with any sequence lengthTwo reductions per token and a learned shiftKernel fusion or quantisation is the priority
No normalisationFewer operations and parametersActivation scale drifts with depthThe model has more than a few layers
Trade-off mapInteractive

Good for

Stable deep training, no warmup requirement

Cost

Slightly weaker final quality at matched compute in some reports

Fails when

You need the last increment of quality and can afford tuning

  1. You need the last increment of quality and can afford tuning
  2. Depth is large and the schedule is short
  3. Kernel fusion or quantisation is the priority
  4. The model has more than a few layers

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Pre-norm is good for Stable deep training, no warmup requirement, costs Slightly weaker final quality at matched compute in some reports, and fails when You need the last increment of quality and can afford tuning. Post-norm is good for Strong final quality, original transformer form, costs Warmup and initialisation sensitivity, and fails when Depth is large and the schedule is short. LayerNorm is good for Batch-independent, works with any sequence length, costs Two reductions per token and a learned shift, and fails when Kernel fusion or quantisation is the priority. No normalisation is good for Fewer operations and parameters, costs Activation scale drifts with depth, and fails when The model has more than a few layers

Failure modes

  • Residual added after the normalisation. The stream is repeatedly rescaled and the identity path disappears, so deep stacks train slowly. Debug: trace the block's input and output and confirm the add uses the pre-normalisation tensor.
  • Normalising over the wrong axis. Statistics are computed across the batch or sequence instead of features, coupling examples and positions. Debug: check that the normalised tensor has zero mean along the feature axis for a single token.
  • Epsilon too small for the dtype. Variance near zero produces huge values or infinities in half precision. Debug: search for non-finite activations after each block on a batch with near-constant inputs.

In production

Log the norm of the residual stream at a few depths on a fixed batch; a steady climb across layers is an early sign of instability. When quantising, keep the normalisation in higher precision, because its reduction is the most sensitive operation in the block.

Common misconceptions

People think: Residual connections mainly help by adding capacity.

Actually: They add no parameters; their role is to provide a gradient path and let blocks learn increments, which is why very deep models need them.

People think: LayerNorm and batch norm are interchangeable normalisations.

Actually: Batch norm uses statistics across examples, so it couples the batch and behaves differently at inference; LayerNorm is per token and batch-independent.

Interview angle

A strong answer explains the identity gradient path, distinguishes pre- and post-norm on where normalisation sits, and says what that does to optimisation. It should note that LayerNorm is batch-independent. Weak answers say residuals 'help gradients flow' without being able to say why normalisation placement changes warmup needs.

Sources