Skip to content
AI Engineer Guide.
midReportedcurated

Why does scaled dot-product attention divide by √d_k?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

What is QK-norm and why do some recent models use it?
  • It normalises each head's query and key vectors before the dot product, which bounds their norms and stops logit magnitude drifting with activation scale; it targets the same saturation the 1/√d_k factor softens and is applied alongside that scaling, not instead of it.
  • It holds up when head dimension or activation scale grows, without needing the extra reduction.
  • The price is an extra reduction per head in the attention path plus implementation complexity, which is why the scalar factor remains the default.
Would saturation get worse with larger or smaller head dimensions?
  • Worse with larger: variance grows with d_k and logits with √d_k, so wide heads saturate the softmax sooner without scaling.
  • Smaller head dimensions shrink the logits, which makes saturation less likely but also narrows the subspace each head can represent.
  • The 1/√d_k factor keeps logit variance roughly constant across those widths, so the softmax stays in the same regime.

Red flags

  • Says the scaling normalizes the output vectors
  • Knows the formula but cannot explain what breaks without it

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals