Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 19 of 33

Multi-head attention

0 of 33 complete

Why it matters

A single attention pattern per layer forces one compromise: the same weighting has to serve syntax, coreference, position and copying at once. Multi-head attention relaxes that constraint by running several attention operations in parallel on lower-dimensional projections, then combining the results.

The design also lines up with hardware. Each head is an independent matrix multiply, so heads parallelise across GPU cores and, at larger scale, across devices through tensor parallelism. That is why head count appears in serving configuration as well as in model architecture.

The practical reason to understand it is debugging and cost. Most multi-head bugs are reshape bugs, and the choice between multi-head, multi-query and grouped-query attention is really a choice about key/value memory during decoding.

Mental model: a panel of readers

Hand the same document to four readers and ask each a different question: one tracks pronouns, one tracks syntax, one tracks nearby words, one tracks the section heading. Each writes a short note. You staple the notes together and read the combined summary.

That captures two real properties: the heads see the same input through different projections, and their outputs are concatenated rather than voted on.

The analogy overstates specialisation. Heads are not assigned roles; whatever division of labour appears is emergent and often redundant. Studies that prune heads find many can be removed with little effect on most inputs, while a few matter a great deal. Treat "this head does syntax" as a hypothesis to test, not a fact.

How it works

Four heads · where the last token looksInteractive

hover or focus a cell

00.55
head (query = mat) ↓key position →
ThecatsatonmatΣ
head 11.000
head 21.000
head 31.000
head 41.000

Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.

Step 1 of 5

Four heads at once

All heads run in the same layer, each with its own score matrix, mask and softmax. They do not read each other's output; concatenation and W_O mix them afterwards.

Next: Head 1

Step 1/5 · Four heads at once

Figure. One layer with four heads, illustrative weights. Each row is a head's attention distribution from the final token, mat, over the five positions; every row totals 1.000. The patterns differ, but this is one invented example, not evidence that heads have fixed jobs.

What to notice. The page's warning applies to the picture: a head that looks local here is not a 'local head' in general, and pruning studies find many heads can be removed with little effect. Read the rows as four simultaneous distributions over the same five keys — that is the mechanical claim.

Text version

A four by five matrix. Rows are heads one to four, columns the tokens The, cat, sat, on and mat. Each cell prints the weight the final token, mat, places on that position, shaded darker for a larger weight, and the row-total column reads 1.000 on all four rows. Head one concentrates on mat at 0.400 and the position before it at 0.360; head two on The at 0.550; head three is spread with a maximum of 0.260; head four splits between The at 0.400 and mat at 0.350. Steps highlight one row at a time, beginning with all four heads, and the readout reports the head, key position and printed weight.

  1. Choose the split. With model width dd and hh heads, each head works in dk=d/hd_k = d / h dimensions. The width must divide evenly.
  2. Project per head. Each head has its own WQW_Q, WKW_K, WVW_V. Implementations usually fuse them into one matrix of shape (d,3d)(d, 3d) and split the output, which is faster than three separate multiplies.
  3. Reshape into heads. Turn (batch, n, d) into (batch, h, n, d_k). The transpose matters: heads must be a separate axis, not interleaved with sequence positions.
  4. Run attention per head. Each head computes its own score matrix, mask, softmax and weighted sum. Heads do not communicate at this stage.
  5. Concatenate. Stack the head outputs back to (batch, n, d) by reversing the reshape.
  6. Mix with the output projection. WOW_O lets the layer combine information across heads; without it, heads would remain separate channels.
  7. Add to the residual. The result joins the residual stream and the next block reads from it.
  8. Change the sharing for efficiency. Multi-query attention shares one key/value head across all queries; grouped-query attention shares within groups. Query heads keep their full count.

The math

Each head is an independent attention operation, and the layer concatenates then projects:

headi=Attention ⁣(XWiQ, XWiK, XWiV),\mathrm{head}_i = \mathrm{Attention}\!\left(XW_i^Q,\, XW_i^K,\, XW_i^V\right), MultiHead(X)=Concat(head1,…,headh) WO.\mathrm{MultiHead}(X) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h)\, W_O.

With dk=d/hd_k = d/h, the projections cost 3d23d^2 and the output projection d2d^2, so the layer holds about 4d24d^2 parameters whether you use four heads or thirty-two. Compute per token is similarly unchanged at fixed total width; only the shape of the parallelism changes.

A minimal implementation

import torch
import torch.nn as nn
 
class MultiHeadAttention(nn.Module):
    def __init__(self, d_model: int = 64, heads: int = 4):
        super().__init__()
        assert d_model % heads == 0, "width must divide evenly"
        self.h, self.d_head = heads, d_model // heads
        self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)   # fused projections
        self.out = nn.Linear(d_model, d_model, bias=False)
 
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        b, n, d = x.shape
        q, k, v = self.qkv(x).chunk(3, dim=-1)                   # each: (b, n, d)
        q, k, v = (t.view(b, n, self.h, self.d_head).transpose(1, 2)
                   for t in (q, k, v))                           # (b, h, n, d_head)
 
        scores = q @ k.transpose(-2, -1) / self.d_head ** 0.5     # (b, h, n, n)
        mask = torch.triu(torch.ones(n, n, dtype=torch.bool), diagonal=1)
        weights = scores.masked_fill(mask, float("-inf")).softmax(dim=-1)
 
        ctx = (weights @ v).transpose(1, 2).reshape(b, n, d)      # concat heads
        return self.out(ctx)
 
mha = MultiHeadAttention(d_model=64, heads=4)
print(mha(torch.randn(2, 10, 64)).shape)          # (2, 10, 64)
print(sum(p.numel() for p in mha.parameters()))   # 4 * 64 * 64 = 16384

The generator expression is convenient but easy to misread; write it as three explicit lines when debugging. transpose(1, 2) before reshape is mandatory, and the reverse order matters on the way back.

Trade-offs

ChoiceGood forCostFails when
One headSmall models, simple tasksOne relation per layerThe layer must track several relationships at once
8–32 headsStandard transformer widthsEach head gets a narrower subspaceHead dimension becomes too small to be useful
Many small headsParallelism, diverse patternsRedundant heads, less capacity eachHardware prefers fewer, larger matrix multiplies
MQA / GQALong context, high concurrencyShared K/V limits head diversityQuality is sensitive and the cache is not the bottleneck
Head pruningCheaper inference after trainingNeeds per-task evaluationA pruned head matters only on rare inputs
Trade-off mapInteractive

Good for

Small models, simple tasks

Cost

One relation per layer

Fails when

The layer must track several relationships at once

  1. The layer must track several relationships at once
  2. Head dimension becomes too small to be useful
  3. Hardware prefers fewer, larger matrix multiplies
  4. Quality is sensitive and the cache is not the bottleneck
  5. A pruned head matters only on rare inputs

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: One head is good for Small models, simple tasks, costs One relation per layer, and fails when The layer must track several relationships at once. 8–32 heads is good for Standard transformer widths, costs Each head gets a narrower subspace, and fails when Head dimension becomes too small to be useful. Many small heads is good for Parallelism, diverse patterns, costs Redundant heads, less capacity each, and fails when Hardware prefers fewer, larger matrix multiplies. MQA / GQA is good for Long context, high concurrency, costs Shared K/V limits head diversity, and fails when Quality is sensitive and the cache is not the bottleneck. Head pruning is good for Cheaper inference after training, costs Needs per-task evaluation, and fails when A pruned head matters only on rare inputs

Failure modes

Head/sequence axis swap. A reshape mixes tokens across heads, so each "head" sees a scrambled sequence. Debugging step: permute the input tokens and check that the output permutes the same way; if it does not, the axes are wrong.

Width not divisible by head count. Some frameworks pad silently, producing heads with unexpected dimensions. Debugging step: assert the division and print d_head for every layer at model load.

Pruning without re-evaluation. Heads that look useless on one benchmark carry rare behaviour — long-range copying, for example. Debugging step: measure the pruned model on task-specific slices, not just average loss.

Reading specialisation into heads. A "pronoun head" is observed on one dataset and assumed everywhere. Debugging step: test the claimed behaviour with controlled examples, and check whether ablating the head changes the output.

In production

Head count is a sharding unit. Tensor parallelism splits heads across devices, so the head count constrains how evenly a model can be distributed. A width that divides badly leaves devices idle.

Head count also affects cache arithmetic once you move to grouped-query or multi-query attention: the query heads stay, but the key/value heads shrink, and decode memory follows the key/value count. This is the change that made long-context serving practical, not a change to the attention maths.

For interpretability work, keep the caveat in view. Attention maps are a useful diagnostic, but they are one layer's routing weights, and many equivalent routings produce the same output.

Common misconceptions

People think: Each attention head has a fixed linguistic job.

Actually: Specialisation is emergent and partial. Pruning studies show many heads can be removed with little effect, while a few matter disproportionately.

People think: More heads means more parameters and more compute.

Actually: At fixed model width, total parameters and FLOPs stay roughly constant; head count changes how the width is partitioned, not how much of it exists.

A third: that concatenation is a vote. Concatenation preserves every head's output as separate channels; the output projection decides how to mix them, and it is learned.

Interview angle

Expect a shapes question: given width and head count, what is each head's dimension, what does the score matrix look like, and where does concatenation happen? Then a cost question: why does doubling heads not double parameters, and what does change when key/value heads are shared? Strong answers connect the arithmetic to tensor parallelism or cache size. See Q/K/V and GQA and MQA, then /interview.

Sources