juniorReportedcurated
Explain self-attention step by step. What exactly are Q, K, and V, and why are K and V separate?
30-second answer
- Each token's vector is projected three ways by learned matrices: a query that says what it is looking for, a key that says what it offers for others to match, and a value that carries what it contributes.
- Scores are QKᵀ divided by √d_k, masked so position i cannot read the future, softmaxed along the key axis so each row sums to one, then multiplied by V and projected back.
- K and V stay separate because findability and content are different jobs; one shared vector would have to do both, and quality measurably drops when it does.
- The price is three projection matrices instead of two plus a larger KV cache, since keys and values are both stored for every cached token.
- Attention only moves information between positions; the feed-forward block at each position does the transform, so attention on its own does not hold the model's stored associations.
Likely follow-ups
Open a question to rehearse the answer.
Why divide scores by √d_k?
- If q and k components are roughly independent with unit variance, their dot product has variance about d_k, so unscaled logits grow like √d_k as heads get wider.
- Large logits drive the softmax toward one-hot, and the p(1−p) terms in its Jacobian vanish at saturation, so the gradient through attention nearly disappears.
- Dividing by √d_k holds logit variance near one regardless of head size, which is a training-dynamics fix rather than a mathematical requirement.
- QK-norm and logit soft-capping attack the same failure harder, at the cost of extra compute and implementation complexity.
In autoregressive decoding, which of Q, K, V get cached and why?
- Keys and values are cached for every layer and every previous position; the query is not, because each new token's query is used once and then discarded.
- The cached K/V vectors do not change once computed, so caching turns a quadratic recompute of the prefix into one projection of the new token.
- Memory follows 2 × layers × KV heads × head dimension × tokens × bytes, and it multiplies by concurrent requests, which is what makes long-context serving expensive.
- Cache validity depends on exact prefix identity, so editing the system prompt or tool list invalidates everything after the change.
Red flags
- Says Q, K and V are three different inputs rather than projections
- Cannot state the softmax dimension or the causal mask step
Concepts this tests
Tags
Source: github.com