midReportedcurated
What does the feed-forward network do, where do most parameters live, and why is SwiGLU the default?
30-second answer
- Attention mixes positions; the feed-forward network transforms each position on its own, and it holds about two-thirds of a dense model's parameters.
- A classic FFN expands to roughly 4× d_model, applies GELU, and projects back, which is where much of the model's stored associations sit.
- SwiGLU replaces that with a gated pair, SiLU(xW_gate) ⊙ (xW_up) followed by W_down, and cuts the hidden size to about 8/3× d_model so the parameter count stays level.
- It wins empirically at matched compute rather than by theorem, and it costs three matrices and more memory traffic than the two-matrix version.
- Because the FFN carries the capacity, it is the part mixture-of-experts replaces, while the attention block and its cache are untouched by that swap.
Likely follow-ups
Open a question to rehearse the answer.
Where does attention overtake the FFN in FLOPs?
- Per token, attention scales with sequence length times width and the FFN with width squared, so the two cross over when the sequence gets long relative to the model width.
- At short context the 8d² MLP dominates; at long context the n² score matrix and its memory traffic dominate, which is the regime fused kernels were built for.
- That crossover is why long prompts cost more prefill before the first token exists, while at short context per-token time in decode is dominated by reading the weights.
Why does MoE swap the FFN rather than attention?
- The FFN is where the parameters and stored associations live, so adding experts there adds capacity without changing the attention maths.
- Attention already spans the sequence and its KV cache is the serving bottleneck, so multiplying attention heads multiplies memory rather than capability.
- The swap leaves attention and the KV cache untouched, so it does not reduce the memory or bandwidth that dominates long-context decode; what it reduces is weight traffic per token, because only the routed experts are read.
Red flags
- Thinks the FFN is a small add-on to attention
- Claims SwiGLU is theoretically optimal rather than empirical
Concepts this tests
Tags
Source: github.com