Why it matters
Attention needs a differentiable lookup: given a query, find the most relevant entries and copy their content. The dot product is the cheapest learned similarity, but its magnitude grows with dimension, which pushes the softmax into saturation where gradients vanish. Dividing by the square root of the key width keeps scores in a range where the distribution stays soft enough to train, which is why the scaling is part of the definition rather than a tuning knob.
Mental model
It is a weighted average where the weights come from a similarity vote: each query votes for keys, votes are normalised, and the result is a blend of values. The metaphor breaks because votes are not independent opinions — queries and keys are learned projections trained to make the blend useful for the final loss, not to be interpretable.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value. Shade and printed number encode the same quantity.
- d_k
- 16
- scale
- ÷ √16 = 4
- score range
- 4.4 – 12.4
Step 1 of 4
Raw QKᵀ scores
Dot products of 16-dimensional vectors in this example. The spread is wide: the largest score is about three times the smallest, so the softmax that follows is close to a hard argmax.
Next: Divide by √d_k
Figure. Six tokens, scores from an illustrative 16-dimensional head. Rows are queries, columns keys. The first frame prints raw QKᵀ dot products; later frames divide by √16 = 4 and then apply softmax. The ordering inside every row is unchanged by scaling; only the spread is.
What to notice. Only the scale changes in the first two frames: every raw score is divided by four, so no key moves rank. The last two frames show why that matters. The unscaled softmax is almost one-hot, with a largest weight of 0.914; the scaled one keeps several keys alive, with a largest weight of 0.399. The two score frames are shaded over the printed range — 4.4–12.4 raw, 1.1–3.1 scaled. The weight frames are shaded against the full 0–1 probability scale, so their top endpoint is the ceiling rather than a value any cell reaches; the largest drawn weight is 0.914.
Text version
A six by six score matrix for The, cat, sat, on, the and mat. The first frame prints raw dot products from 4.4 to 12.4. The second frame divides every cell by four, giving 1.1 to 3.1, and the row ordering is identical. The third frame applies softmax after scaling: cells print weights, the largest is 0.399, and every row total is 1.000. The fourth frame applies softmax without scaling: the largest weight rises to 0.914 and the other keys fall near zero. Steps switch between these frames and the readout reports the query, key and printed value.
- Project the input into queries, keys and values, each with its own learned matrix and head dimension.
- Compute the score matrix as queries times keys transposed, giving one score per query-key pair.
- Divide the scores by the square root of the key dimension to keep their variance roughly constant.
- Add the mask, using a large negative value for positions that must not be attended.
- Apply softmax along the key axis so each row becomes a probability distribution.
- Multiply by the values and project the result, then pass it to the residual stream.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Full pairwise scores | Exact attention, simple to reason about | Quadratic time and score memory in sequence length | Contexts are long or batches are large |
| Fused IO-aware kernels | Same mathematics with far less memory traffic | Backend and dtype constraints, harder debugging | The head dimension or device is unsupported |
| Low-precision scores | Faster matmuls, smaller score buffers | Softmax sensitivity to small logit changes | The task needs fine discrimination among close keys |
| Head dimension larger than 64 | More expressive per-head similarity | Scale factor and kernel support change | The serving kernel only supports standard widths |
Good for
Exact attention, simple to reason about
Cost
Quadratic time and score memory in sequence length
Fails when
Contexts are long or batches are large
- Contexts are long or batches are large
- The head dimension or device is unsupported
- The task needs fine discrimination among close keys
- The serving kernel only supports standard widths
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Full pairwise scores is good for Exact attention, simple to reason about, costs Quadratic time and score memory in sequence length, and fails when Contexts are long or batches are large. Fused IO-aware kernels is good for Same mathematics with far less memory traffic, costs Backend and dtype constraints, harder debugging, and fails when The head dimension or device is unsupported. Low-precision scores is good for Faster matmuls, smaller score buffers, costs Softmax sensitivity to small logit changes, and fails when The task needs fine discrimination among close keys. Head dimension larger than 64 is good for More expressive per-head similarity, costs Scale factor and kernel support change, and fails when The serving kernel only supports standard widths
Failure modes
- Missing scale factor. Scores grow with dimension, softmax collapses onto one key and gradients vanish. Debug: log the entropy of the attention distribution; a value near zero across heads points at missing or wrong scaling.
- Softmax on the wrong axis. Normalising over queries instead of keys makes each key's weights sum to one and destroys the lookup. Debug: assert that rows of the weight matrix sum to one along the key dimension.
- Mask added after softmax. Forbidden positions still receive probability mass and leak information. Debug: set a future token to a distinct value and confirm earlier outputs are bit-identical.
In production
Watch attention entropy and the fraction of mass on the first token, since both drift when scaling or masking is wrong. For long contexts, measure peak memory for the score matrix or confirm that a fused kernel avoids materialising it.
Common misconceptions
People think: The square-root scaling is arbitrary tuning.
Actually: It compensates for the variance of a dot product over d dimensions; removing it makes the softmax saturate as width grows.
People think: Attention is expensive because of the softmax.
Actually: The softmax is elementwise and cheap; the quadratic score matrix and its memory traffic dominate.
Interview angle
A strong answer writes the formula, explains each step and justifies the scale factor from variance, then discusses masking and the memory cost of the score matrix. It should mention fused kernels as an implementation of the same math. Weak answers recite the formula without being able to say why the denominator exists or where the quadratic term lives.
Sources
- Vaswani et al., Attention Is All You Need — the scaled dot-product formulation and complexity analysis.
- Dao et al., FlashAttention — computing exact attention without materialising the score matrix.
- Harvard NLP, The Annotated Transformer — a line-by-line reference implementation.