Why it matters
During decode, every generated token must read the entire key-value cache. With one key-value head per query head, that cache is large and the read dominates step time. Sharing key-value heads reduces the bytes moved per step and lets more requests fit in memory. The design question is how much head diversity to give up, because each shared head removes a distinct set of keys that queries could otherwise specialise on.
Mental model
It is a newsroom where several reporters share one research file instead of each keeping a private one: less filing, faster lookup, but less independent sourcing. The metaphor breaks because the shared file is not merely smaller — the projections that produced it are trained jointly, so sharing changes what the keys represent.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.
- query heads
- 8
- KV heads
- 4
- per group
- 8 ÷ 4 = 2
- KV cache
- 64 KiB per token
Step 1 of 4
The partition
Every query head sits in exactly one group, so each row has a single 1. No head is split across KV heads and no group is empty, which is what the divisibility check protects.
Next: Group kv₁
Figure. Grouped-query attention drawn as a partition, illustrative layout. Each row is a query head and each column a key-value head; a 1 means that query head reads that KV head and a 0 that it does not. Eight query heads over four KV heads is two query heads per group, so every row totals 1 and every column totals 2. The divisibility is the gate: a head count that does not divide cannot be laid out this way.
What to notice. The saving lives in the columns, not the rows. Keys and values are cached per KV head, so with 32 layers, head_dim 128 and fp16 one token costs 2 × 32 × 4 × 128 × 2 bytes = 64 KiB at four KV heads, 128 KiB at eight (multi-head) and 16 KiB at one (multi-query). Queries keep their full row count, so the quality cost comes from fewer distinct key sets, not from fewer queries. The figure omits the projections and the repeat that expands a shared head across its group.
Text version
An eight by four membership matrix. Rows are query heads q₁ to q₈, columns are KV heads kv₁ to kv₄, and a cell prints 1 when the query head reads that KV head and 0 when it does not. q₁ and q₂ print 1 in the kv₁ column and 0 elsewhere; q₃ and q₄ belong to kv₂; q₅ and q₆ to kv₃; q₇ and q₈ to kv₄. Every row-total cell reads 1 and every column total reads 2. Cells shade from 0 to 1 on one scale. Steps highlight the eight membership cells, the kv₁ column and the first two rows; the column-total row is always visible. The readout reports the query head, the KV head and the membership value.
- Count query heads and decide how many key-value heads to keep, ensuring the query count divides evenly by the key-value count.
- Project keys and values with fewer output channels, then repeat or expand them across the query groups at attention time.
- Keep queries at full head count so each query head still has its own projection and its own attention pattern.
- Train or uptrain with the grouped configuration, converting existing key-value projections by mean-pooling within each group.
- Verify that the cache layout in the serving kernel supports the head ratio, since not every kernel handles arbitrary groupings.
- Measure quality on long-context and retrieval-heavy tasks, where reduced key diversity is most likely to show.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Multi-head (one KV per query) | Maximum head diversity and quality | Largest cache, highest decode bandwidth | Concurrency or context length is memory-bound |
| Grouped-query | Most of the quality with a much smaller cache | Some quality loss, kernel support varies | The task depends on fine-grained multi-head retrieval |
| Multi-query | Smallest cache, fastest decode | Noticeable quality loss in some settings | Quality is the priority |
| Uptraining from multi-head | Cheap conversion of an existing checkpoint | Requires additional training and evaluation | You cannot afford any further training |
Good for
Maximum head diversity and quality
Cost
Largest cache, highest decode bandwidth
Fails when
Concurrency or context length is memory-bound
- Concurrency or context length is memory-bound
- The task depends on fine-grained multi-head retrieval
- Quality is the priority
- You cannot afford any further training
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Multi-head (one KV per query) is good for Maximum head diversity and quality, costs Largest cache, highest decode bandwidth, and fails when Concurrency or context length is memory-bound. Grouped-query is good for Most of the quality with a much smaller cache, costs Some quality loss, kernel support varies, and fails when The task depends on fine-grained multi-head retrieval. Multi-query is good for Smallest cache, fastest decode, costs Noticeable quality loss in some settings, and fails when Quality is the priority. Uptraining from multi-head is good for Cheap conversion of an existing checkpoint, costs Requires additional training and evaluation, and fails when You cannot afford any further training
Failure modes
- Head ratio not divisible. Expanding key-value heads across query groups leaves a remainder and the kernel silently truncates or errors. Debug: assert query heads modulo key-value heads equals zero at model construction.
- Cache allocated for the wrong head count. The server reserves multi-head-sized cache for a grouped model and wastes the memory it was meant to save. Debug: compute cache bytes from the model config and compare against the allocator's report.
- Quality regression on long context. Sharing keys hurts tasks that need to distinguish many positions at once. Debug: evaluate the grouped model on long-context retrieval slices against the multi-head baseline.
In production
Report cache memory per request and maximum concurrent sequences from the actual head configuration, not from a nameplate number. Re-run long-context evaluations after any head-sharing change, since aggregate benchmarks often miss the regression.
Common misconceptions
People think: Grouped-query attention speeds up the attention matmul.
Actually: It mainly reduces cache size and memory traffic during decode; the arithmetic saving is secondary to the bandwidth saving.
People think: Sharing key-value heads is free quality-wise.
Actually: Each shared head reduces the distinct key sets available to queries, and the loss appears on tasks that need fine-grained discrimination.
Interview angle
A strong answer draws the head layout, explains the cache arithmetic and why decode is the phase that benefits, and names the quality cost. It should mention uptraining and kernel constraints. Weak answers call it 'an efficiency trick' without saying what is shared or which phase of inference changes.
Sources
- Shazeer, Fast Transformer Decoding: One Write-Head is All You Need — multi-query attention and decode-time cache cost.
- Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — grouped-query attention and uptraining from multi-head.
- Dao, FlashAttention-2 — how head layout and cache traffic affect attention kernels.