juniorReportedcurated
Walk me through what happens inside a single transformer decoder block.
30-second answer
- A decoder block is two residual sublayers: causal multi-head attention that mixes information across positions, then a position-wise MLP that transforms each position independently.
- Llama-style blocks pre-norm each sublayer with RMSNorm, apply RoPE to Q and K inside attention, and put the norm before the sublayer so the residual path stays an identity.
- GPT-2 and OPT use LayerNorm in the same pre-norm position, and the original transformer places the norm after each sublayer, which is post-norm and needs warm-up to train at depth.
- Per-layer parameters split roughly 4d² for attention and 8d² for the MLP, so the MLP holds about two-thirds of each block's parameters before the embedding matrices are counted.
- A final norm and the LM head turn the last hidden state into vocabulary logits, and depth buys capability at the cost of per-layer latency and memory.
Likely follow-ups
Open a question to rehearse the answer.
What fraction of a block's parameters live in the MLP versus attention?
- At width d with a 4d expansion, attention holds about 4d² and the feed-forward about 8d², so the MLP is two-thirds of the block by parameter count.
- The split is a parameter fact, not a compute fact: at short sequences the MLP already dominates FLOPs per token, and attention only catches up as context grows.
- That is why mixture-of-experts replaces the MLP, and why a model can add capacity there without touching the attention maths.
What changes in this picture for a mixture-of-experts model?
- The attention sublayer stays as it is; the dense MLP is replaced by several expert MLPs plus a router that sends each token to a few of them.
- Total parameters grow with the number of experts while per-token compute stays near a dense model's, so you buy capacity without proportional FLOPs.
- The costs move to routing, load balancing and holding every expert in memory, and the design fails when traffic is small or batches are tiny.
- It does not fix attention cost or the KV cache, which are unchanged by the swap.
Red flags
- Puts LayerNorm on the residual path without knowing it is post-norm
- Says attention stores knowledge and the MLP only mixes positions
Concepts this tests
Tags
Source: github.com