Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 3 of 33

MLP

0 of 33 complete

Why it matters

Attention alone is a weighted average, and averaging cannot compute new features. A transformer needs a per-position transformation with more capacity than a single linear map, but it cannot afford position-specific weights because sequence length varies. The MLP solves both: it applies one shared two-layer network to every position, adding nonlinear feature construction without coupling positions or growing with context.

Mental model

The MLP is a per-token workshop: each token arrives with a bag of features, gets expanded into a larger workspace, is recombined nonlinearly, then compressed back to the standard width. The metaphor breaks because the workshop has no memory — the identical tooling is applied to every token, so any token-specific behaviour must already be encoded in its input vector.

How it works

Per-token MLP: one vector widened and narrowedInteractive
signal

Step 1 of 5

One position at a time

Each token's 4096-wide vector enters on its own. The MLP has no path between positions, so order never enters this sublayer.

Active layers

  • Token vector

Hover a layer to read what it holds.

Next: Widen

W₁
67.1M parameters
W₂
67.1M parameters
Per block
≈134M · illustrative
Step 1/5 · One position at a time

Figure. The feed-forward network applied at a single position. W₁ widens 4096 features to 16384, GELU acts elementwise, and W₂ narrows back to 4096; the same two matrices are reused at every position. The 4× ratio is the common convention, not a requirement.

What to notice. No edge in this stack crosses positions: attention is the only sublayer that moves information between tokens. Parameter count is set by width, not sequence length, so the block costs the same at 8 tokens or 8,000. The 67.1M-per-matrix figure assumes d_model = 4096 and a 4× expansion.

Text version

A five-band stack. A token vector of width 4096 feeds an up-projection matrix of shape 4096 by 16384, then GELU at width 16384, then a down-projection of shape 16384 by 4096, and leaves at width 4096. Five steps describe the per-position scope, the parameter count of each matrix, the nonlinearity, and the fact that the weights are shared across all positions.

  1. Take the residual-stream vector at one position, of width d_model, as the sublayer input.
  2. Project upward to a wider hidden size, conventionally a multiple of d_model, using a learned matrix.
  3. Apply the nonlinearity. GELU in most modern blocks, ReLU in older ones — to the expanded vector.
  4. Project back down to d_model so the result can be added to the residual stream.
  5. Apply the identical weights at every position, which is why the block has no notion of order on its own.
  6. Replace the plain two-matrix form with a gated variant when quality matters more than FLOPs, since the gate adds a third matrix.

Trade-offs

ChoiceGood forCostFails when
4x expansion, two matricesStandard quality and simple kernelsTwo-thirds of block parameters and FLOPsMemory or latency is the hard constraint
Smaller expansionCheaper per token, smaller checkpointsLess feature capacity per layerThe task needs compositional transformations
Gated MLP (SwiGLU)Better quality at matched compute in many reportsA third weight matrix and more activation memoryThe serving stack cannot fuse the extra matmul
Mixture-of-experts MLPMore capacity without proportional computeRouting complexity and memory for all expertsTraffic is small or batch sizes are tiny
Trade-off mapInteractive

Good for

Standard quality and simple kernels

Cost

Two-thirds of block parameters and FLOPs

Fails when

Memory or latency is the hard constraint

  1. Memory or latency is the hard constraint
  2. The task needs compositional transformations
  3. The serving stack cannot fuse the extra matmul
  4. Traffic is small or batch sizes are tiny

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: 4x expansion, two matrices is good for Standard quality and simple kernels, costs Two-thirds of block parameters and FLOPs, and fails when Memory or latency is the hard constraint. Smaller expansion is good for Cheaper per token, smaller checkpoints, costs Less feature capacity per layer, and fails when The task needs compositional transformations. Gated MLP (SwiGLU) is good for Better quality at matched compute in many reports, costs A third weight matrix and more activation memory, and fails when The serving stack cannot fuse the extra matmul. Mixture-of-experts MLP is good for More capacity without proportional compute, costs Routing complexity and memory for all experts, and fails when Traffic is small or batch sizes are tiny

Failure modes

  • Expansion ratio mistaken for free. Checkpoint and activation memory blow past the plan because the hidden width was raised without re-measuring. Debug: compute parameters and activation bytes from the actual dimensions before launching a run.
  • Nonlinearity dropped during export. A fused or quantised kernel replaces GELU with a cheaper approximation and outputs drift on hard inputs. Debug: compare logits against the reference implementation on a fixed batch after every export step.
  • Position leakage through the MLP. A per-position MLP is expected to be independent, but a reshape bug mixes neighbouring tokens. Debug: perturb one position and assert the MLP output at other positions is unchanged.

In production

Watch per-layer activation memory during long-context training, since the expanded hidden state is what usually forces gradient checkpointing. At serving time, the MLP is often bandwidth-bound during decode, so fusing its matrices matters more than its FLOP count suggests.

Common misconceptions

People think: The MLP is just a formality after attention.

Actually: It performs the nonlinear feature construction that attention cannot: attention reweights existing vectors, while the MLP creates new combinations within a position.

People think: A bigger expansion ratio always improves quality.

Actually: Past a point the extra width adds parameters and latency without helping, and the right ratio depends on depth, data and whether the block uses gating.

Interview angle

A strong answer places the MLP inside the block, gives its shapes, and explains the division of labour with attention: channels within a position versus mixing across positions. It should mention where the parameters live and why the width ratio is a capacity decision. Weak answers describe it as two linear layers without saying why a shared per-position network is the right constraint.

Sources