Why it matters
Every description of attention eventually says "query, key, value" and moves on. That shorthand hides the detail that determines memory, cache design and a class of shape bugs: all three come from the same tensor, and the only difference between them is which weight matrix was applied.
The distinction becomes operational at inference. Keys and values for a prefix never change, so they are cached; queries are recomputed for each new token. Grouped-query attention reduces the number of key and value heads while keeping the query heads, and that choice is legible only if you know which tensor is reused.
Mental model: a library request
You walk into a library with a request — your query. Each shelf has a label describing what it holds — its key. You compare your request with the labels, choose shelves that match, and take the contents — the values — back with you.
The mapping is direct enough to be useful: query is intent, key is advertised content, value is actual content. It also explains why keys and values are usually projected separately: what a passage advertises and what it delivers need not be the same representation.
The analogy breaks because the match is soft: you take a weighted mixture from every shelf, not the single best one. And nobody wrote the labels. Keys are learned projections optimised so the downstream loss improves, so a key may encode something no human would label.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value. Shade and printed number encode the same quantity.
Step 1 of 4
One input, three blocks
Every row is one token. The six columns are output units grouped into Q, K and V. In a real layer each group is a separate learned matrix, drawn together because all three read the same input row.
Next: Query block
Figure. Four tokens, two output units per projection, illustrative activations. Each row is one token; the three column blocks are the query, key and value projections of the same input row. Teal is below zero and accent above; the printed signed value carries the magnitude and the sign.
What to notice. The claim is the grouping and the shared rows, not the numbers. Q, K and V are three learned matrices applied to one tensor; in a real layer they are separate d_model × d_head matrices, often fused into one multiply and split afterwards. The three headers are the only block labels — solid for Q, dashed for K, dotted for V — so hue is reserved for the value: teal below zero, accent above. One sign scale is used across all three blocks, so the blocks are comparable on that scale; the printed value is exact. The activations are invented, so nothing here is evidence about a trained model.
Text version
A four by six matrix for the tokens The, cat, sat and end, with columns grouped into Q, K and V, two dimensions each. The group headers are underlined solid for Q, dashed for K and dotted for V, in ink rather than a value hue. Each cell prints a signed illustrative activation, shaded teal when negative and accent when positive on one shared scale, with values from minus 1.1 to 1.5. Steps highlight the Q block, then K, then V, and the readout reports the token, the output unit and the printed activation. There is no row-total column because the three blocks are separate projections, not parts of one distribution.
- Start from one tensor. The block receives of shape for a sequence of positions.
- Project three times. Apply learned matrices , , , each of shape for a single head. The three outputs are the queries, keys and values.
- Score query–key pairs. produces an matrix: entry is how much position 's query matches position 's key.
- Scale, mask, normalise. Divide by , add the causal or padding mask, then softmax each row.
- Blend values. The weights multiply , so each output position receives a weighted sum of value vectors.
- Project the result. Concatenated head outputs pass through back to so the block output can be added to the residual stream.
- Reuse K and V when decoding. For a new token, compute its query, append its key and value to the cache, and attend over the cached pairs.
- Swap the source for cross-attention. Queries come from the decoder state; keys and values come from the encoder output, so and can differ.
The math
For a single head with input :
The projection parameters per head are . With heads and , that totals for the projections, plus for the output projection. With grouped-query attention, and produce fewer heads, so their parameter and cache cost shrink while stays full size.
A minimal implementation
import torch
import torch.nn as nn
class SingleHeadAttention(nn.Module):
def __init__(self, d_model: int, d_head: int):
super().__init__()
self.d_head = d_head
self.wq = nn.Linear(d_model, d_head, bias=False)
self.wk = nn.Linear(d_model, d_head, bias=False)
self.wv = nn.Linear(d_model, d_head, bias=False)
def forward(self, x, context=None, causal=True):
src = x if context is None else context # cross-attention source
q = self.wq(x) # (batch, n_q, d_head)
k = self.wk(src) # (batch, n_kv, d_head)
v = self.wv(src) # (batch, n_kv, d_head)
scores = q @ k.transpose(-2, -1) / self.d_head ** 0.5 # (batch, n_q, n_kv)
if causal and context is None:
n_q, n_kv = scores.shape[-2:]
mask = torch.triu(torch.ones(n_q, n_kv, dtype=torch.bool), diagonal=1)
scores = scores.masked_fill(mask, float("-inf"))
return scores.softmax(dim=-1) @ v # (batch, n_q, d_head)
attn = SingleHeadAttention(d_model=64, d_head=16)
x = torch.randn(2, 5, 64)
print(attn(x).shape) # torch.Size([2, 5, 16])
print(attn(x, context=torch.randn(2, 9, 64)).shape) # torch.Size([2, 5, 16])
print(sum(p.numel() for p in attn.parameters())) # 3 * 64 * 16 = 3072Two shape facts are worth internalising: the output keeps the query length, and the score matrix is (n_q, n_kv). In self-attention they are equal; in cross-attention they are not, and a mask built from the wrong one breaks without an error.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Separate Q/K/V projections | Default; lets match and content differ | 3d² parameters per layer | Model is tiny and parameter budget dominates |
| Tied or shared projections | Very small models, weight savings | Less expressive matching | The task needs distinct match and content views |
| Bias terms | Some architectures and fine-tuning regimes | Extra parameters, little effect at scale | You assume bias changes behaviour materially |
| Full-size KV heads (MHA) | Maximum expressiveness per head | Cache = full size per token | Long context or high concurrency |
| Fewer KV heads (GQA/MQA) | Cheaper cache and decode bandwidth | Small quality risk, head-sharing constraints | Head dimension does not divide evenly |
Good for
Default; lets match and content differ
Cost
3d² parameters per layer
Fails when
Model is tiny and parameter budget dominates
- Model is tiny and parameter budget dominates
- The task needs distinct match and content views
- You assume bias changes behaviour materially
- Long context or high concurrency
- Head dimension does not divide evenly
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Separate Q/K/V projections is good for Default; lets match and content differ, costs 3d² parameters per layer, and fails when Model is tiny and parameter budget dominates. Tied or shared projections is good for Very small models, weight savings, costs Less expressive matching, and fails when The task needs distinct match and content views. Bias terms is good for Some architectures and fine-tuning regimes, costs Extra parameters, little effect at scale, and fails when You assume bias changes behaviour materially. Full-size KV heads (MHA) is good for Maximum expressiveness per head, costs Cache = full size per token, and fails when Long context or high concurrency. Fewer KV heads (GQA/MQA) is good for Cheaper cache and decode bandwidth, costs Small quality risk, head-sharing constraints, and fails when Head dimension does not divide evenly
Failure modes
Head and model dimensions conflated. A reshape assumes and silently mixes head contents. Debugging step: assert d_model % heads == 0 and print tensor shapes at each projection during the first forward pass.
Masking the wrong tensor. The mask is applied to values or after the blend instead of to the scores. Debugging step: set one forbidden value to a huge number and confirm the output for the masked position is unchanged.
Caching queries. A custom decode loop stores Q alongside K and V, inflating cache memory and reusing a stale query. Debugging step: print cache shapes per layer; Q should not grow with the sequence.
Cross-attention length mismatch. The causal mask is built from the query length and applied to key positions, cutting off valid context. Debugging step: assert the mask shape equals the score shape before the softmax.
In production
Cache size is a direct function of K/V heads, layers, head dimension and sequence length. Reducing KV heads — multi-query or grouped-query attention — is the standard lever, and it trades a little quality for materially lower memory and decode bandwidth.
Prefix caching works because K and V for a shared prefix are identical across requests. If your prompt template is stable, the cached prefix can be reused; if the template changes per request, hit rate collapses. This is a K/V property, not a general "prompt cache" feature.
When profiling, remember that queries are cheap and transient while K/V persist. Most decode-time memory pressure comes from the tensors you keep, not the ones you compute.
Common misconceptions
People think: Query, key and value are three different inputs to attention.
Actually: They are three linear projections of the same input in self-attention. Only in cross-attention do keys and values come from a different source.
People think: Keys are the document and values are the answer, so caching keys is enough.
Actually: Decoding needs both K and V for every cached position. Caching only keys leaves nothing to blend.
A third: that Q/K/V is an analogy bolted onto the maths. The names describe roles in one equation; the model is free to learn projections that do not correspond to any human notion of asking and answering.
Interview angle
Q/K/V questions test whether you can hold shapes and reuse in your head. A good answer distinguishes the three projections, derives (n_q, n_kv) from the multiplication, and explains why decode caches K and V but not Q. The follow-up is usually grouped-query attention: which tensor shrinks, and what that does to memory. Continue with multi-head attention and KV cache, or see /interview.
Sources
- Vaswani et al., Attention Is All You Need — the projection definitions and multi-head formulation: https://arxiv.org/abs/1706.03762
- Harvard NLP, The Annotated Transformer — explicit Q/K/V and mask code: https://nlp.seas.harvard.edu/annotated-transformer/
- Shazeer, Fast Transformer Decoding: One Write-Head is All You Need — multi-query attention and why K/V dominate decode: https://arxiv.org/abs/1911.02150
- Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models — grouping key/value heads as a middle ground: https://arxiv.org/abs/2305.13245