midReportedcurated
Why use multiple attention heads instead of one head of the full model width?
30-second answer
- One softmax per token gives one weighting over positions, which has to serve pronoun resolution, syntax, adjacency and delimiter matching at the same time.
- Multi-head attention splits the width into h subspaces, runs an independent Q/K/V and softmax in each, concatenates the results and mixes them with W_o.
- Parameters stay near 4d² regardless of head count, while per-token attention FLOPs are about 4d² + 4nd, so the compute share shifts with context while each head works in a narrower subspace.
- Heads are partly redundant: pruning studies remove many with little effect on most inputs while a few matter disproportionately, which is what motivates GQA and MQA.
- Head dimensions of 64–128 are the common sweet spot, because too small a subspace cannot represent a useful relation and too few heads removes diversity.
Likely follow-ups
Open a question to rehearse the answer.
How would you test whether a head is doing something specific?
- Ablate it: zero or mean the head's output and compare on controlled inputs where the claimed behaviour should appear.
- Check the claim on inputs you built for it rather than on the benchmark where the correlation was first noticed, because specialisation is often partial and dataset-specific.
- Run the permutation test for axis bugs first, since an output that does not permute with its input tokens is a reshape failure rather than a head behaviour.
- Treat “this head does syntax” as a hypothesis, because many equivalent routings produce the same output.
If many heads are redundant, what serving optimizations does that justify?
- Grouped-query or multi-query attention keeps the query heads but shares K/V heads, which shrinks the cache and the decode read without changing the attention maths.
- Head pruning removes heads outright and needs per-task evaluation, because a head that looks useless on one benchmark can carry rare long-range copying.
- Tensor parallelism already shards by head, so head count constrains how evenly a model can be spread across devices.
- None of these shorten prefill attention or change the quadratic term; they attack cache memory and bandwidth.
Red flags
- Says multiple heads add FLOPs and that is why they are better
- Cannot explain the concat plus W_o step
Concepts this tests
Tags
Source: github.com