Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 18 of 33

Scaled dot-product attention

0 of 33 complete

Why it matters

Attention needs a differentiable lookup: given a query, find the most relevant entries and copy their content. The dot product is the cheapest learned similarity, but its magnitude grows with dimension, which pushes the softmax into saturation where gradients vanish. Dividing by the square root of the key width keeps scores in a range where the distribution stays soft enough to train, which is why the scaling is part of the definition rather than a tuning knob.

Mental model

It is a weighted average where the weights come from a similarity vote: each query votes for keys, votes are normalised, and the result is a blend of values. The metaphor breaks because votes are not independent opinions — queries and keys are learned projections trained to make the blend useful for the final loss, not to be interpretable.

How it works

Scores before and after scalingInteractive

hover or focus a cell

4.412.4
query ↓key →
Thecatsatonthemat
The
cat
sat
on
the
mat

Hover or focus a cell to read its exact value. Shade and printed number encode the same quantity.

d_k
16
scale
÷ √16 = 4
score range
4.4 – 12.4

Step 1 of 4

Raw QKᵀ scores

Dot products of 16-dimensional vectors in this example. The spread is wide: the largest score is about three times the smallest, so the softmax that follows is close to a hard argmax.

Next: Divide by √d_k

Step 1/4 · Raw QKᵀ scores

Figure. Six tokens, scores from an illustrative 16-dimensional head. Rows are queries, columns keys. The first frame prints raw QKᵀ dot products; later frames divide by √16 = 4 and then apply softmax. The ordering inside every row is unchanged by scaling; only the spread is.

What to notice. Only the scale changes in the first two frames: every raw score is divided by four, so no key moves rank. The last two frames show why that matters. The unscaled softmax is almost one-hot, with a largest weight of 0.914; the scaled one keeps several keys alive, with a largest weight of 0.399. The two score frames are shaded over the printed range — 4.4–12.4 raw, 1.1–3.1 scaled. The weight frames are shaded against the full 0–1 probability scale, so their top endpoint is the ceiling rather than a value any cell reaches; the largest drawn weight is 0.914.

Text version

A six by six score matrix for The, cat, sat, on, the and mat. The first frame prints raw dot products from 4.4 to 12.4. The second frame divides every cell by four, giving 1.1 to 3.1, and the row ordering is identical. The third frame applies softmax after scaling: cells print weights, the largest is 0.399, and every row total is 1.000. The fourth frame applies softmax without scaling: the largest weight rises to 0.914 and the other keys fall near zero. Steps switch between these frames and the readout reports the query, key and printed value.

  1. Project the input into queries, keys and values, each with its own learned matrix and head dimension.
  2. Compute the score matrix as queries times keys transposed, giving one score per query-key pair.
  3. Divide the scores by the square root of the key dimension to keep their variance roughly constant.
  4. Add the mask, using a large negative value for positions that must not be attended.
  5. Apply softmax along the key axis so each row becomes a probability distribution.
  6. Multiply by the values and project the result, then pass it to the residual stream.

Trade-offs

ChoiceGood forCostFails when
Full pairwise scoresExact attention, simple to reason aboutQuadratic time and score memory in sequence lengthContexts are long or batches are large
Fused IO-aware kernelsSame mathematics with far less memory trafficBackend and dtype constraints, harder debuggingThe head dimension or device is unsupported
Low-precision scoresFaster matmuls, smaller score buffersSoftmax sensitivity to small logit changesThe task needs fine discrimination among close keys
Head dimension larger than 64More expressive per-head similarityScale factor and kernel support changeThe serving kernel only supports standard widths
Trade-off mapInteractive

Good for

Exact attention, simple to reason about

Cost

Quadratic time and score memory in sequence length

Fails when

Contexts are long or batches are large

  1. Contexts are long or batches are large
  2. The head dimension or device is unsupported
  3. The task needs fine discrimination among close keys
  4. The serving kernel only supports standard widths

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full pairwise scores is good for Exact attention, simple to reason about, costs Quadratic time and score memory in sequence length, and fails when Contexts are long or batches are large. Fused IO-aware kernels is good for Same mathematics with far less memory traffic, costs Backend and dtype constraints, harder debugging, and fails when The head dimension or device is unsupported. Low-precision scores is good for Faster matmuls, smaller score buffers, costs Softmax sensitivity to small logit changes, and fails when The task needs fine discrimination among close keys. Head dimension larger than 64 is good for More expressive per-head similarity, costs Scale factor and kernel support change, and fails when The serving kernel only supports standard widths

Failure modes

  • Missing scale factor. Scores grow with dimension, softmax collapses onto one key and gradients vanish. Debug: log the entropy of the attention distribution; a value near zero across heads points at missing or wrong scaling.
  • Softmax on the wrong axis. Normalising over queries instead of keys makes each key's weights sum to one and destroys the lookup. Debug: assert that rows of the weight matrix sum to one along the key dimension.
  • Mask added after softmax. Forbidden positions still receive probability mass and leak information. Debug: set a future token to a distinct value and confirm earlier outputs are bit-identical.

In production

Watch attention entropy and the fraction of mass on the first token, since both drift when scaling or masking is wrong. For long contexts, measure peak memory for the score matrix or confirm that a fused kernel avoids materialising it.

Common misconceptions

People think: The square-root scaling is arbitrary tuning.

Actually: It compensates for the variance of a dot product over d dimensions; removing it makes the softmax saturate as width grows.

People think: Attention is expensive because of the softmax.

Actually: The softmax is elementwise and cheap; the quadratic score matrix and its memory traffic dominate.

Interview angle

A strong answer writes the formula, explains each step and justifies the scale factor from variance, then discusses masking and the memory cost of the score matrix. It should mention fused kernels as an implementation of the same math. Weak answers recite the formula without being able to say why the denominator exists or where the quadratic term lives.

Sources