Why it matters
A single attention pattern per layer forces one compromise: the same weighting has to serve syntax, coreference, position and copying at once. Multi-head attention relaxes that constraint by running several attention operations in parallel on lower-dimensional projections, then combining the results.
The design also lines up with hardware. Each head is an independent matrix multiply, so heads parallelise across GPU cores and, at larger scale, across devices through tensor parallelism. That is why head count appears in serving configuration as well as in model architecture.
The practical reason to understand it is debugging and cost. Most multi-head bugs are reshape bugs, and the choice between multi-head, multi-query and grouped-query attention is really a choice about key/value memory during decoding.
Mental model: a panel of readers
Hand the same document to four readers and ask each a different question: one tracks pronouns, one tracks syntax, one tracks nearby words, one tracks the section heading. Each writes a short note. You staple the notes together and read the combined summary.
That captures two real properties: the heads see the same input through different projections, and their outputs are concatenated rather than voted on.
The analogy overstates specialisation. Heads are not assigned roles; whatever division of labour appears is emergent and often redundant. Studies that prune heads find many can be removed with little effect on most inputs, while a few matter a great deal. Treat "this head does syntax" as a hypothesis to test, not a fact.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.
Step 1 of 5
Four heads at once
All heads run in the same layer, each with its own score matrix, mask and softmax. They do not read each other's output; concatenation and W_O mix them afterwards.
Next: Head 1
Figure. One layer with four heads, illustrative weights. Each row is a head's attention distribution from the final token, mat, over the five positions; every row totals 1.000. The patterns differ, but this is one invented example, not evidence that heads have fixed jobs.
What to notice. The page's warning applies to the picture: a head that looks local here is not a 'local head' in general, and pruning studies find many heads can be removed with little effect. Read the rows as four simultaneous distributions over the same five keys — that is the mechanical claim.
Text version
A four by five matrix. Rows are heads one to four, columns the tokens The, cat, sat, on and mat. Each cell prints the weight the final token, mat, places on that position, shaded darker for a larger weight, and the row-total column reads 1.000 on all four rows. Head one concentrates on mat at 0.400 and the position before it at 0.360; head two on The at 0.550; head three is spread with a maximum of 0.260; head four splits between The at 0.400 and mat at 0.350. Steps highlight one row at a time, beginning with all four heads, and the readout reports the head, key position and printed weight.
- Choose the split. With model width and heads, each head works in dimensions. The width must divide evenly.
- Project per head. Each head has its own , , . Implementations usually fuse them into one matrix of shape and split the output, which is faster than three separate multiplies.
- Reshape into heads. Turn
(batch, n, d)into(batch, h, n, d_k). The transpose matters: heads must be a separate axis, not interleaved with sequence positions. - Run attention per head. Each head computes its own score matrix, mask, softmax and weighted sum. Heads do not communicate at this stage.
- Concatenate. Stack the head outputs back to
(batch, n, d)by reversing the reshape. - Mix with the output projection. lets the layer combine information across heads; without it, heads would remain separate channels.
- Add to the residual. The result joins the residual stream and the next block reads from it.
- Change the sharing for efficiency. Multi-query attention shares one key/value head across all queries; grouped-query attention shares within groups. Query heads keep their full count.
The math
Each head is an independent attention operation, and the layer concatenates then projects:
With , the projections cost and the output projection , so the layer holds about parameters whether you use four heads or thirty-two. Compute per token is similarly unchanged at fixed total width; only the shape of the parallelism changes.
A minimal implementation
import torch
import torch.nn as nn
class MultiHeadAttention(nn.Module):
def __init__(self, d_model: int = 64, heads: int = 4):
super().__init__()
assert d_model % heads == 0, "width must divide evenly"
self.h, self.d_head = heads, d_model // heads
self.qkv = nn.Linear(d_model, 3 * d_model, bias=False) # fused projections
self.out = nn.Linear(d_model, d_model, bias=False)
def forward(self, x: torch.Tensor) -> torch.Tensor:
b, n, d = x.shape
q, k, v = self.qkv(x).chunk(3, dim=-1) # each: (b, n, d)
q, k, v = (t.view(b, n, self.h, self.d_head).transpose(1, 2)
for t in (q, k, v)) # (b, h, n, d_head)
scores = q @ k.transpose(-2, -1) / self.d_head ** 0.5 # (b, h, n, n)
mask = torch.triu(torch.ones(n, n, dtype=torch.bool), diagonal=1)
weights = scores.masked_fill(mask, float("-inf")).softmax(dim=-1)
ctx = (weights @ v).transpose(1, 2).reshape(b, n, d) # concat heads
return self.out(ctx)
mha = MultiHeadAttention(d_model=64, heads=4)
print(mha(torch.randn(2, 10, 64)).shape) # (2, 10, 64)
print(sum(p.numel() for p in mha.parameters())) # 4 * 64 * 64 = 16384The generator expression is convenient but easy to misread; write it as three explicit lines when debugging. transpose(1, 2) before reshape is mandatory, and the reverse order matters on the way back.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| One head | Small models, simple tasks | One relation per layer | The layer must track several relationships at once |
| 8–32 heads | Standard transformer widths | Each head gets a narrower subspace | Head dimension becomes too small to be useful |
| Many small heads | Parallelism, diverse patterns | Redundant heads, less capacity each | Hardware prefers fewer, larger matrix multiplies |
| MQA / GQA | Long context, high concurrency | Shared K/V limits head diversity | Quality is sensitive and the cache is not the bottleneck |
| Head pruning | Cheaper inference after training | Needs per-task evaluation | A pruned head matters only on rare inputs |
Good for
Small models, simple tasks
Cost
One relation per layer
Fails when
The layer must track several relationships at once
- The layer must track several relationships at once
- Head dimension becomes too small to be useful
- Hardware prefers fewer, larger matrix multiplies
- Quality is sensitive and the cache is not the bottleneck
- A pruned head matters only on rare inputs
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: One head is good for Small models, simple tasks, costs One relation per layer, and fails when The layer must track several relationships at once. 8–32 heads is good for Standard transformer widths, costs Each head gets a narrower subspace, and fails when Head dimension becomes too small to be useful. Many small heads is good for Parallelism, diverse patterns, costs Redundant heads, less capacity each, and fails when Hardware prefers fewer, larger matrix multiplies. MQA / GQA is good for Long context, high concurrency, costs Shared K/V limits head diversity, and fails when Quality is sensitive and the cache is not the bottleneck. Head pruning is good for Cheaper inference after training, costs Needs per-task evaluation, and fails when A pruned head matters only on rare inputs
Failure modes
Head/sequence axis swap. A reshape mixes tokens across heads, so each "head" sees a scrambled sequence. Debugging step: permute the input tokens and check that the output permutes the same way; if it does not, the axes are wrong.
Width not divisible by head count. Some frameworks pad silently, producing heads with unexpected dimensions. Debugging step: assert the division and print d_head for every layer at model load.
Pruning without re-evaluation. Heads that look useless on one benchmark carry rare behaviour — long-range copying, for example. Debugging step: measure the pruned model on task-specific slices, not just average loss.
Reading specialisation into heads. A "pronoun head" is observed on one dataset and assumed everywhere. Debugging step: test the claimed behaviour with controlled examples, and check whether ablating the head changes the output.
In production
Head count is a sharding unit. Tensor parallelism splits heads across devices, so the head count constrains how evenly a model can be distributed. A width that divides badly leaves devices idle.
Head count also affects cache arithmetic once you move to grouped-query or multi-query attention: the query heads stay, but the key/value heads shrink, and decode memory follows the key/value count. This is the change that made long-context serving practical, not a change to the attention maths.
For interpretability work, keep the caveat in view. Attention maps are a useful diagnostic, but they are one layer's routing weights, and many equivalent routings produce the same output.
Common misconceptions
People think: Each attention head has a fixed linguistic job.
Actually: Specialisation is emergent and partial. Pruning studies show many heads can be removed with little effect, while a few matter disproportionately.
People think: More heads means more parameters and more compute.
Actually: At fixed model width, total parameters and FLOPs stay roughly constant; head count changes how the width is partitioned, not how much of it exists.
A third: that concatenation is a vote. Concatenation preserves every head's output as separate channels; the output projection decides how to mix them, and it is learned.
Interview angle
Expect a shapes question: given width and head count, what is each head's dimension, what does the score matrix look like, and where does concatenation happen? Then a cost question: why does doubling heads not double parameters, and what does change when key/value heads are shared? Strong answers connect the arithmetic to tensor parallelism or cache size. See Q/K/V and GQA and MQA, then /interview.
Sources
- Vaswani et al., Attention Is All You Need — the multi-head formulation and per-head dimension choice: https://arxiv.org/abs/1706.03762
- Michel, Levy & Neubig, Are Sixteen Heads Really Better than One? — evidence on head redundancy and pruning: https://arxiv.org/abs/1905.10650
- Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models — sharing key/value heads while keeping query heads: https://arxiv.org/abs/2305.13245
- Harvard NLP, The Annotated Transformer — reference implementation of head splitting and concatenation: https://nlp.seas.harvard.edu/annotated-transformer/