Skip to content
AI Engineer Guide.
midReportedcurated

Why use multiple attention heads instead of one head of the full model width?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How would you test whether a head is doing something specific?
  • Ablate it: zero or mean the head's output and compare on controlled inputs where the claimed behaviour should appear.
  • Check the claim on inputs you built for it rather than on the benchmark where the correlation was first noticed, because specialisation is often partial and dataset-specific.
  • Run the permutation test for axis bugs first, since an output that does not permute with its input tokens is a reshape failure rather than a head behaviour.
  • Treat “this head does syntax” as a hypothesis, because many equivalent routings produce the same output.
If many heads are redundant, what serving optimizations does that justify?
  • Grouped-query or multi-query attention keeps the query heads but shares K/V heads, which shrinks the cache and the decode read without changing the attention maths.
  • Head pruning removes heads outright and needs per-task evaluation, because a head that looks useless on one benchmark can carry rare long-range copying.
  • Tensor parallelism already shards by head, so head count constrains how evenly a model can be spread across devices.
  • None of these shorten prefill attention or change the quadratic term; they attack cache memory and bandwidth.

Red flags

  • Says multiple heads add FLOPs and that is why they are better
  • Cannot explain the concat plus W_o step

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals