midReportedcurated
Why does scaled dot-product attention divide by √d_k?
30-second answer
- If q and k components are roughly independent with unit variance, their dot product has variance about d_k, so unscaled logits grow like √d_k as the head dimension rises.
- Large logits push the softmax toward one-hot, and the p(1−p) terms in its Jacobian vanish at saturation, so the gradient through attention nearly disappears.
- Dividing by √d_k holds logit variance near one independent of head size, which keeps the distribution soft enough to train.
- It is a training-dynamics fix rather than a mathematical requirement: the attention operation is still correct without it, it just becomes hard to optimise.
- QK-norm and logit soft-capping attack the same failure more aggressively, at the cost of extra compute and implementation complexity.
Likely follow-ups
Open a question to rehearse the answer.
What is QK-norm and why do some recent models use it?
- It normalises each head's query and key vectors before the dot product, which bounds their norms and stops logit magnitude drifting with activation scale; it targets the same saturation the 1/√d_k factor softens and is applied alongside that scaling, not instead of it.
- It holds up when head dimension or activation scale grows, without needing the extra reduction.
- The price is an extra reduction per head in the attention path plus implementation complexity, which is why the scalar factor remains the default.
Would saturation get worse with larger or smaller head dimensions?
- Worse with larger: variance grows with d_k and logits with √d_k, so wide heads saturate the softmax sooner without scaling.
- Smaller head dimensions shrink the logits, which makes saturation less likely but also narrows the subspace each head can represent.
- The 1/√d_k factor keeps logit variance roughly constant across those widths, so the softmax stays in the same regime.
Red flags
- Says the scaling normalizes the output vectors
- Knows the formula but cannot explain what breaks without it
Concepts this tests
Tags
Source: github.com