Why it matters
Attention can only reweight vectors that already exist; it cannot build a new feature from a combination of channels. The feed-forward block supplies that construction with a nonlinearity, but it must do so without position-specific parameters, or the model would not transfer across lengths. Sharing one small network across positions is the compromise that gives capacity without coupling to sequence layout.
Mental model
The block is a standardised tool head that every workstation uses on its own part: it widens the part, reshapes it, then narrows it back to the line's gauge. The metaphor breaks because the tool head is not applied in sequence along the line — every position is processed in parallel with the same weights and no communication between stations.
How it works
Step 1 of 6
Copy the stream
The block reads the residual stream at one position. Every position goes through the same weights in parallel, so nothing here depends on sequence order.
Active layers
- Residual stream
Hover a layer to read what it holds.
Next: Normalise the branch
- Widths
- d 4096 · 4× illustrative
- Parameters
- ≈134M per block
- Mixing
- within position only
Figure. One position's vector passes through the pre-norm feed-forward block while the residual edge carries the unchanged input to the add. The 4× expansion and 4096-wide model are illustrative; the arrangement is the claim.
What to notice. No band in this stack mixes positions: every operation runs on the same single vector, which is why attention and the feed-forward block have different jobs. The residual edge skips the whole branch — normalisation and all three transforms — so the block contributes an increment rather than a replacement. At d_model = 4096 with a 4× expansion the two matrices hold about 134M parameters per block.
Text version
A vertical stack. The residual stream x enters and feeds a pre-norm LayerNorm, then an up-projection from d_model to 4 times d_model, then GELU at the wider size, then a down-projection back to d_model. That output goes to a residual add, and the original x bypasses the whole block along a right-hand edge into the same add. The block output leaves at d_model. Six steps walk copy, normalise, expand, activate, project back, and add.
- Normalise the residual-stream vector at each position before the block, in the pre-norm arrangement.
- Project from the model width to an expanded hidden width with the first weight matrix.
- Apply the nonlinearity, or split the projection into two and multiply one branch by the activated other in a gated variant.
- Project back to the model width with the second matrix so the output matches the residual stream.
- Add the result to the stream, leaving the block's contribution as an increment rather than a replacement.
- Check the expansion ratio against memory and latency budgets, since this block usually dominates both.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Plain two-matrix FFN | Simple, well-supported kernels | Less quality per parameter than gated forms | You are chasing the last increment of quality |
| Gated FFN | Better quality at matched compute in many settings | Third matrix, more activation memory | Memory or kernel support is tight |
| Wide expansion | More per-position capacity | Parameter and FLOP growth, slower decode | Serving cost is the constraint |
| Narrow expansion with more layers | Depth-wise composition | More sequential steps, harder optimisation | Latency is dominated by layer count |
Good for
Simple, well-supported kernels
Cost
Less quality per parameter than gated forms
Fails when
You are chasing the last increment of quality
- You are chasing the last increment of quality
- Memory or kernel support is tight
- Serving cost is the constraint
- Latency is dominated by layer count
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Plain two-matrix FFN is good for Simple, well-supported kernels, costs Less quality per parameter than gated forms, and fails when You are chasing the last increment of quality. Gated FFN is good for Better quality at matched compute in many settings, costs Third matrix, more activation memory, and fails when Memory or kernel support is tight. Wide expansion is good for More per-position capacity, costs Parameter and FLOP growth, slower decode, and fails when Serving cost is the constraint. Narrow expansion with more layers is good for Depth-wise composition, costs More sequential steps, harder optimisation, and fails when Latency is dominated by layer count
Failure modes
- Expansion dimension mismatched to the head. The final projection does not return to model width and the residual add fails or broadcasts. Debug: assert the block output shape equals the block input shape for one batch.
- Activation applied to the wrong branch. A gated variant multiplies before the nonlinearity, changing the function and hurting quality. Debug: compare the block against a reference implementation on fixed weights.
- FFN dominating decode latency. The block reads its full weight matrices per token and becomes the bottleneck. Debug: profile per-module time during decode and compare against attention time.
In production
Track the share of step time spent in the feed-forward block; during decode it is usually bandwidth-bound, so quantising or fusing these matrices pays more than tuning attention. Re-measure after changing the expansion ratio, since parameter count and activation memory both move.
Common misconceptions
People think: The feed-forward block is a minor add-on after attention.
Actually: It holds most of the parameters and performs the nonlinear feature construction that attention cannot, so removing it removes most of the model's capacity.
People think: Widening the FFN is a cheap way to add capacity.
Actually: It increases parameters, activation memory and per-token bandwidth cost, and the benefit saturates relative to adding data or depth.
Interview angle
A strong answer distinguishes per-position channel mixing from cross-position attention, gives the shapes and the expansion ratio, and explains where the parameters sit. It should mention gating and the decode-time bandwidth cost. Weak answers call it 'just an MLP' without saying why it is shared across positions or why it dominates serving cost.
Sources
- Shazeer, GLU Variants Improve Transformer — gated variants and their empirical effect.
- Geva et al., Transformer Feed-Forward Layers Are Key-Value Memories — an interpretation of what the block stores.
- Alammar, The Illustrated Transformer — where the feed-forward block sits in the layer.