Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 17 of 33

Q/K/V

0 of 33 complete

Why it matters

Every description of attention eventually says "query, key, value" and moves on. That shorthand hides the detail that determines memory, cache design and a class of shape bugs: all three come from the same tensor, and the only difference between them is which weight matrix was applied.

The distinction becomes operational at inference. Keys and values for a prefix never change, so they are cached; queries are recomputed for each new token. Grouped-query attention reduces the number of key and value heads while keeping the query heads, and that choice is legible only if you know which tensor is reused.

Mental model: a library request

You walk into a library with a request — your query. Each shelf has a label describing what it holds — its key. You compare your request with the labels, choose shelves that match, and take the contents — the values — back with you.

The mapping is direct enough to be useful: query is intent, key is advertised content, value is actual content. It also explains why keys and values are usually projected separately: what a passage advertises and what it delivers need not be the same representation.

The analogy breaks because the match is soft: you take a weighted mixture from every shelf, not the single best one. And nobody wrote the labels. Keys are learned projections optimised so the downstream loss improves, so a key may encode something no human would label.

How it works

Q, K and V · one input, three projection blocksInteractive

hover or focus a cell

−1.5+1.5midpoint 0
token ↓output unit →
QKV
q₁q₂k₁k₂v₁v₂
The
cat
sat
end

Hover or focus a cell to read its exact value. Shade and printed number encode the same quantity.

Step 1 of 4

One input, three blocks

Every row is one token. The six columns are output units grouped into Q, K and V. In a real layer each group is a separate learned matrix, drawn together because all three read the same input row.

Next: Query block

Step 1/4 · One input, three blocks

Figure. Four tokens, two output units per projection, illustrative activations. Each row is one token; the three column blocks are the query, key and value projections of the same input row. Teal is below zero and accent above; the printed signed value carries the magnitude and the sign.

What to notice. The claim is the grouping and the shared rows, not the numbers. Q, K and V are three learned matrices applied to one tensor; in a real layer they are separate d_model × d_head matrices, often fused into one multiply and split afterwards. The three headers are the only block labels — solid for Q, dashed for K, dotted for V — so hue is reserved for the value: teal below zero, accent above. One sign scale is used across all three blocks, so the blocks are comparable on that scale; the printed value is exact. The activations are invented, so nothing here is evidence about a trained model.

Text version

A four by six matrix for the tokens The, cat, sat and end, with columns grouped into Q, K and V, two dimensions each. The group headers are underlined solid for Q, dashed for K and dotted for V, in ink rather than a value hue. Each cell prints a signed illustrative activation, shaded teal when negative and accent when positive on one shared scale, with values from minus 1.1 to 1.5. Steps highlight the Q block, then K, then V, and the readout reports the token, the output unit and the printed activation. There is no row-total column because the three blocks are separate projections, not parts of one distribution.

  1. Start from one tensor. The block receives XX of shape (n,dmodel)(n, d_{\text{model}}) for a sequence of nn positions.
  2. Project three times. Apply learned matrices WQW_Q, WKW_K, WVW_V, each of shape (dmodel,dhead)(d_{\text{model}}, d_{\text{head}}) for a single head. The three outputs are the queries, keys and values.
  3. Score query–key pairs. QK⊤QK^\top produces an (n,n)(n, n) matrix: entry (i,j)(i, j) is how much position ii's query matches position jj's key.
  4. Scale, mask, normalise. Divide by dhead\sqrt{d_{\text{head}}}, add the causal or padding mask, then softmax each row.
  5. Blend values. The weights multiply VV, so each output position receives a weighted sum of value vectors.
  6. Project the result. Concatenated head outputs pass through WOW_O back to dmodeld_{\text{model}} so the block output can be added to the residual stream.
  7. Reuse K and V when decoding. For a new token, compute its query, append its key and value to the cache, and attend over the cached pairs.
  8. Swap the source for cross-attention. Queries come from the decoder state; keys and values come from the encoder output, so nqn_q and nkvn_{kv} can differ.

The math

For a single head with input X∈Rn×dX \in \mathbb{R}^{n \times d}:

Q=XWQ,K=XWK,V=XWV,WQ,WK,WV∈Rd×dk,Q = XW_Q,\quad K = XW_K,\quad V = XW_V, \qquad W_Q, W_K, W_V \in \mathbb{R}^{d \times d_k}, head=softmax ⁣(QK⊤dk)V∈Rn×dk.\mathrm{head} = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V \in \mathbb{R}^{n \times d_k}.

The projection parameters per head are 3⋅d⋅dk3 \cdot d \cdot d_k. With hh heads and dk=d/hd_k = d/h, that totals 3d23d^2 for the projections, plus d2d^2 for the output projection. With grouped-query attention, WKW_K and WVW_V produce fewer heads, so their parameter and cache cost shrink while WQW_Q stays full size.

A minimal implementation

import torch
import torch.nn as nn
 
class SingleHeadAttention(nn.Module):
    def __init__(self, d_model: int, d_head: int):
        super().__init__()
        self.d_head = d_head
        self.wq = nn.Linear(d_model, d_head, bias=False)
        self.wk = nn.Linear(d_model, d_head, bias=False)
        self.wv = nn.Linear(d_model, d_head, bias=False)
 
    def forward(self, x, context=None, causal=True):
        src = x if context is None else context      # cross-attention source
        q = self.wq(x)                               # (batch, n_q, d_head)
        k = self.wk(src)                             # (batch, n_kv, d_head)
        v = self.wv(src)                             # (batch, n_kv, d_head)
 
        scores = q @ k.transpose(-2, -1) / self.d_head ** 0.5   # (batch, n_q, n_kv)
        if causal and context is None:
            n_q, n_kv = scores.shape[-2:]
            mask = torch.triu(torch.ones(n_q, n_kv, dtype=torch.bool), diagonal=1)
            scores = scores.masked_fill(mask, float("-inf"))
        return scores.softmax(dim=-1) @ v            # (batch, n_q, d_head)
 
attn = SingleHeadAttention(d_model=64, d_head=16)
x = torch.randn(2, 5, 64)
print(attn(x).shape)                                 # torch.Size([2, 5, 16])
print(attn(x, context=torch.randn(2, 9, 64)).shape)  # torch.Size([2, 5, 16])
print(sum(p.numel() for p in attn.parameters()))     # 3 * 64 * 16 = 3072

Two shape facts are worth internalising: the output keeps the query length, and the score matrix is (n_q, n_kv). In self-attention they are equal; in cross-attention they are not, and a mask built from the wrong one breaks without an error.

Trade-offs

ChoiceGood forCostFails when
Separate Q/K/V projectionsDefault; lets match and content differ3d² parameters per layerModel is tiny and parameter budget dominates
Tied or shared projectionsVery small models, weight savingsLess expressive matchingThe task needs distinct match and content views
Bias termsSome architectures and fine-tuning regimesExtra parameters, little effect at scaleYou assume bias changes behaviour materially
Full-size KV heads (MHA)Maximum expressiveness per headCache = full size per tokenLong context or high concurrency
Fewer KV heads (GQA/MQA)Cheaper cache and decode bandwidthSmall quality risk, head-sharing constraintsHead dimension does not divide evenly
Trade-off mapInteractive

Good for

Default; lets match and content differ

Cost

3d² parameters per layer

Fails when

Model is tiny and parameter budget dominates

  1. Model is tiny and parameter budget dominates
  2. The task needs distinct match and content views
  3. You assume bias changes behaviour materially
  4. Long context or high concurrency
  5. Head dimension does not divide evenly

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Separate Q/K/V projections is good for Default; lets match and content differ, costs 3d² parameters per layer, and fails when Model is tiny and parameter budget dominates. Tied or shared projections is good for Very small models, weight savings, costs Less expressive matching, and fails when The task needs distinct match and content views. Bias terms is good for Some architectures and fine-tuning regimes, costs Extra parameters, little effect at scale, and fails when You assume bias changes behaviour materially. Full-size KV heads (MHA) is good for Maximum expressiveness per head, costs Cache = full size per token, and fails when Long context or high concurrency. Fewer KV heads (GQA/MQA) is good for Cheaper cache and decode bandwidth, costs Small quality risk, head-sharing constraints, and fails when Head dimension does not divide evenly

Failure modes

Head and model dimensions conflated. A reshape assumes dhead=dmodeld_{\text{head}} = d_{\text{model}} and silently mixes head contents. Debugging step: assert d_model % heads == 0 and print tensor shapes at each projection during the first forward pass.

Masking the wrong tensor. The mask is applied to values or after the blend instead of to the scores. Debugging step: set one forbidden value to a huge number and confirm the output for the masked position is unchanged.

Caching queries. A custom decode loop stores Q alongside K and V, inflating cache memory and reusing a stale query. Debugging step: print cache shapes per layer; Q should not grow with the sequence.

Cross-attention length mismatch. The causal mask is built from the query length and applied to key positions, cutting off valid context. Debugging step: assert the mask shape equals the score shape before the softmax.

In production

Cache size is a direct function of K/V heads, layers, head dimension and sequence length. Reducing KV heads — multi-query or grouped-query attention — is the standard lever, and it trades a little quality for materially lower memory and decode bandwidth.

Prefix caching works because K and V for a shared prefix are identical across requests. If your prompt template is stable, the cached prefix can be reused; if the template changes per request, hit rate collapses. This is a K/V property, not a general "prompt cache" feature.

When profiling, remember that queries are cheap and transient while K/V persist. Most decode-time memory pressure comes from the tensors you keep, not the ones you compute.

Common misconceptions

People think: Query, key and value are three different inputs to attention.

Actually: They are three linear projections of the same input in self-attention. Only in cross-attention do keys and values come from a different source.

People think: Keys are the document and values are the answer, so caching keys is enough.

Actually: Decoding needs both K and V for every cached position. Caching only keys leaves nothing to blend.

A third: that Q/K/V is an analogy bolted onto the maths. The names describe roles in one equation; the model is free to learn projections that do not correspond to any human notion of asking and answering.

Interview angle

Q/K/V questions test whether you can hold shapes and reuse in your head. A good answer distinguishes the three projections, derives (n_q, n_kv) from the multiplication, and explains why decode caches K and V but not Q. The follow-up is usually grouped-query attention: which tensor shrinks, and what that does to memory. Continue with multi-head attention and KV cache, or see /interview.

Sources