Skip to content
AI Engineer Guide.
midReportedcurated

What does the feed-forward network do, where do most parameters live, and why is SwiGLU the default?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Where does attention overtake the FFN in FLOPs?
  • Per token, attention scales with sequence length times width and the FFN with width squared, so the two cross over when the sequence gets long relative to the model width.
  • At short context the 8d² MLP dominates; at long context the n² score matrix and its memory traffic dominate, which is the regime fused kernels were built for.
  • That crossover is why long prompts cost more prefill before the first token exists, while at short context per-token time in decode is dominated by reading the weights.
Why does MoE swap the FFN rather than attention?
  • The FFN is where the parameters and stored associations live, so adding experts there adds capacity without changing the attention maths.
  • Attention already spans the sequence and its KV cache is the serving bottleneck, so multiplying attention heads multiplies memory rather than capability.
  • The swap leaves attention and the KV cache untouched, so it does not reduce the memory or bandwidth that dominates long-context decode; what it reduces is weight traffic per token, because only the routed experts are read.

Red flags

  • Thinks the FFN is a small add-on to attention
  • Claims SwiGLU is theoretically optimal rather than empirical

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals