Skip to content
AI Engineer Guide.
juniorReportedcurated

Walk me through what happens inside a single transformer decoder block.

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

What fraction of a block's parameters live in the MLP versus attention?
  • At width d with a 4d expansion, attention holds about 4d² and the feed-forward about 8d², so the MLP is two-thirds of the block by parameter count.
  • The split is a parameter fact, not a compute fact: at short sequences the MLP already dominates FLOPs per token, and attention only catches up as context grows.
  • That is why mixture-of-experts replaces the MLP, and why a model can add capacity there without touching the attention maths.
What changes in this picture for a mixture-of-experts model?
  • The attention sublayer stays as it is; the dense MLP is replaced by several expert MLPs plus a router that sends each token to a few of them.
  • Total parameters grow with the number of experts while per-token compute stays near a dense model's, so you buy capacity without proportional FLOPs.
  • The costs move to routing, load balancing and holding every expert in memory, and the design fails when traffic is small or batches are tiny.
  • It does not fix attention cost or the KV cache, which are unchanged by the swap.

Red flags

  • Puts LayerNorm on the residual path without knowing it is post-norm
  • Says attention stores knowledge and the MLP only mixes positions

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals