Skip to content
AI Engineer Guide.
juniorReportedcurated

Explain self-attention step by step. What exactly are Q, K, and V, and why are K and V separate?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why divide scores by √d_k?
  • If q and k components are roughly independent with unit variance, their dot product has variance about d_k, so unscaled logits grow like √d_k as heads get wider.
  • Large logits drive the softmax toward one-hot, and the p(1−p) terms in its Jacobian vanish at saturation, so the gradient through attention nearly disappears.
  • Dividing by √d_k holds logit variance near one regardless of head size, which is a training-dynamics fix rather than a mathematical requirement.
  • QK-norm and logit soft-capping attack the same failure harder, at the cost of extra compute and implementation complexity.
In autoregressive decoding, which of Q, K, V get cached and why?
  • Keys and values are cached for every layer and every previous position; the query is not, because each new token's query is used once and then discarded.
  • The cached K/V vectors do not change once computed, so caching turns a quadratic recompute of the prefix into one projection of the new token.
  • Memory follows 2 × layers × KV heads × head dimension × tokens × bytes, and it multiplies by concurrent requests, which is what makes long-context serving expensive.
  • Cache validity depends on exact prefix identity, so editing the system prompt or tool list invalidates everything after the change.

Red flags

  • Says Q, K and V are three different inputs rather than projections
  • Cannot state the softmax dimension or the causal mask step

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals