Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 23 of 33

Feed-forward block

0 of 33 complete

Why it matters

Attention can only reweight vectors that already exist; it cannot build a new feature from a combination of channels. The feed-forward block supplies that construction with a nonlinearity, but it must do so without position-specific parameters, or the model would not transfer across lengths. Sharing one small network across positions is the compromise that gives capacity without coupling to sequence layout.

Mental model

The block is a standardised tool head that every workstation uses on its own part: it widens the part, reshapes it, then narrows it back to the line's gauge. The metaphor breaks because the tool head is not applied in sequence along the line — every position is processed in parallel with the same weights and no communication between stations.

How it works

Inside the block: expand, activate, project, addInteractive
signalresidual

Step 1 of 6

Copy the stream

The block reads the residual stream at one position. Every position goes through the same weights in parallel, so nothing here depends on sequence order.

Active layers

  • Residual stream

Hover a layer to read what it holds.

Next: Normalise the branch

Widths
d 4096 · 4× illustrative
Parameters
≈134M per block
Mixing
within position only
Step 1/6 · Copy the stream

Figure. One position's vector passes through the pre-norm feed-forward block while the residual edge carries the unchanged input to the add. The 4× expansion and 4096-wide model are illustrative; the arrangement is the claim.

What to notice. No band in this stack mixes positions: every operation runs on the same single vector, which is why attention and the feed-forward block have different jobs. The residual edge skips the whole branch — normalisation and all three transforms — so the block contributes an increment rather than a replacement. At d_model = 4096 with a 4× expansion the two matrices hold about 134M parameters per block.

Text version

A vertical stack. The residual stream x enters and feeds a pre-norm LayerNorm, then an up-projection from d_model to 4 times d_model, then GELU at the wider size, then a down-projection back to d_model. That output goes to a residual add, and the original x bypasses the whole block along a right-hand edge into the same add. The block output leaves at d_model. Six steps walk copy, normalise, expand, activate, project back, and add.

  1. Normalise the residual-stream vector at each position before the block, in the pre-norm arrangement.
  2. Project from the model width to an expanded hidden width with the first weight matrix.
  3. Apply the nonlinearity, or split the projection into two and multiply one branch by the activated other in a gated variant.
  4. Project back to the model width with the second matrix so the output matches the residual stream.
  5. Add the result to the stream, leaving the block's contribution as an increment rather than a replacement.
  6. Check the expansion ratio against memory and latency budgets, since this block usually dominates both.

Trade-offs

ChoiceGood forCostFails when
Plain two-matrix FFNSimple, well-supported kernelsLess quality per parameter than gated formsYou are chasing the last increment of quality
Gated FFNBetter quality at matched compute in many settingsThird matrix, more activation memoryMemory or kernel support is tight
Wide expansionMore per-position capacityParameter and FLOP growth, slower decodeServing cost is the constraint
Narrow expansion with more layersDepth-wise compositionMore sequential steps, harder optimisationLatency is dominated by layer count
Trade-off mapInteractive

Good for

Simple, well-supported kernels

Cost

Less quality per parameter than gated forms

Fails when

You are chasing the last increment of quality

  1. You are chasing the last increment of quality
  2. Memory or kernel support is tight
  3. Serving cost is the constraint
  4. Latency is dominated by layer count

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Plain two-matrix FFN is good for Simple, well-supported kernels, costs Less quality per parameter than gated forms, and fails when You are chasing the last increment of quality. Gated FFN is good for Better quality at matched compute in many settings, costs Third matrix, more activation memory, and fails when Memory or kernel support is tight. Wide expansion is good for More per-position capacity, costs Parameter and FLOP growth, slower decode, and fails when Serving cost is the constraint. Narrow expansion with more layers is good for Depth-wise composition, costs More sequential steps, harder optimisation, and fails when Latency is dominated by layer count

Failure modes

  • Expansion dimension mismatched to the head. The final projection does not return to model width and the residual add fails or broadcasts. Debug: assert the block output shape equals the block input shape for one batch.
  • Activation applied to the wrong branch. A gated variant multiplies before the nonlinearity, changing the function and hurting quality. Debug: compare the block against a reference implementation on fixed weights.
  • FFN dominating decode latency. The block reads its full weight matrices per token and becomes the bottleneck. Debug: profile per-module time during decode and compare against attention time.

In production

Track the share of step time spent in the feed-forward block; during decode it is usually bandwidth-bound, so quantising or fusing these matrices pays more than tuning attention. Re-measure after changing the expansion ratio, since parameter count and activation memory both move.

Common misconceptions

People think: The feed-forward block is a minor add-on after attention.

Actually: It holds most of the parameters and performs the nonlinear feature construction that attention cannot, so removing it removes most of the model's capacity.

People think: Widening the FFN is a cheap way to add capacity.

Actually: It increases parameters, activation memory and per-token bandwidth cost, and the benefit saturates relative to adding data or depth.

Interview angle

A strong answer distinguishes per-position channel mixing from cross-position attention, gives the shapes and the expansion ratio, and explains where the parameters sit. It should mention gating and the decode-time bandwidth cost. Weak answers call it 'just an MLP' without saying why it is shared across positions or why it dominates serving cost.

Sources