Why it matters
Self-attention is the mechanism that makes a language model context-sensitive. The same token gets a different representation depending on what surrounds it: "bank" near "river" and "bank" near "loan" end up in different places in vector space, because each position reads from its neighbours and reweights itself accordingly.
It is also where the cost lives. Attention compares every position with every other position, so doubling the context quadruples the pairwise work and the score matrix. Long-context pricing, the KV cache, grouped-query attention and flash kernels are all responses to that one quadratic term.
Understanding the mechanism at the level of shapes and masks is what lets you debug the failures: future-token leakage, padding that is attended as content, or a context that is technically accepted but effectively ignored.
Mental model: a soft dictionary lookup
Each position asks a question (its query), every position advertises what it contains (its key), and every position carries content (its value). The query is compared with every key, the scores are normalised into weights, and the position receives a weighted blend of values.
Retrieval engineers recognise this immediately: it is a search over the sequence, with the retrieval weights learned rather than computed by cosine similarity.
The analogy breaks in three ways. The lookup is soft — every key gets some weight, not just the best match. All positions do this simultaneously, in parallel, rather than one query at a time. And the keys and values are not authored descriptions; they are linear projections learned end to end, so what a key "advertises" is whatever helped the training loss.
How it works
- Project the input. From the block's input , compute queries, keys and values with learned weight matrices: , , .
- Score every pair. Multiply by to get an matrix of raw affinities for a sequence of length .
- Scale. Divide by , the square root of the key dimension. Without it, dot products grow with dimension and push softmax into saturation.
- Mask. Add a causal mask for decoder self-attention so position cannot read positions after . Mask padding so pad tokens are not treated as content.
- Normalise. Apply softmax along the key axis, so each row of weights sums to 1.
- Blend the values. Multiply the weights by . Each output row is a weighted sum of value vectors from the positions it attended to.
- Repeat per head and project. Multiple heads run in parallel with their own projections; their outputs are concatenated and mixed by an output projection (see multi-head attention).
- Stack. The result is added to the residual stream and passed to the next block.
The math
where is the additive mask: for allowed positions and for forbidden ones. For a sequence of length and head dimension :
The quadratic term is in , not in the parameter count. That distinction explains why a model with fixed weights becomes progressively more expensive per token as the prompt grows.
A minimal implementation
import numpy as np
def softmax(x: np.ndarray, axis: int = -1) -> np.ndarray:
x = x - x.max(axis=axis, keepdims=True)
e = np.exp(x)
return e / e.sum(axis=axis, keepdims=True)
def self_attention(Q, K, V, causal: bool = True):
d_k = Q.shape[-1]
scores = Q @ np.swapaxes(K, -1, -2) / np.sqrt(d_k) # (..., n, n)
if causal:
n = scores.shape[-1]
future = np.triu(np.ones((n, n), dtype=bool), k=1)
scores = np.where(future, -np.inf, scores)
weights = softmax(scores, axis=-1) # rows sum to 1
return weights @ V, weights
rng = np.random.default_rng(0)
X = rng.normal(size=(1, 4, 8)) # batch, tokens, d_model
Wq, Wk, Wv = (rng.normal(size=(8, 8)) for _ in range(3))
out, w = self_attention(X @ Wq, X @ Wk, X @ Wv)
print(out.shape, w.shape) # (1, 4, 8) (1, 4, 4)
print(w[0, -1].round(2)) # last token may read all four positionsOne trap hides in the mask. If a row is fully masked, max returns -inf, subtraction yields nan, and the whole row becomes nan instead of zeros. Causal masks avoid this because the first position always attends to itself; padding masks with an empty sequence do not.
weightmasked
Row 1 · query “The” reads from itself only.
Illustrative single-head weights, renormalised after masking. Real models run several heads per layer. Toggle the mask off to see what the same head would attend to without it — the future cells fill in, and the causal property disappears.
Figure. One head over six tokens, with illustrative weights. Each row is a query and each column a key; a cell shows the weight that query places on that key, renormalised so the row sums to 1.000. Cells are printed to three decimals with the rounding residue assigned by largest remainder, so the printed row also adds to exactly 1.000.
What to notice. The mask is applied before softmax, not after. Future positions are set to −∞, so they contribute exactly zero once the surviving weights are renormalised; masking after normalisation would leave the row summing to less than 1. The Σ column adds the printed cells, not the unrounded ones.
Source: Vaswani et al., Attention Is All You Need (2017), arXiv:1706.03762. Weights are illustrative, not read from a trained model.
Text version
A six by six attention matrix for the tokens The, cat, sat, on, the and mat. Rows are queries, columns are keys, and each filled cell prints its weight to three decimals and shades it darker for a larger weight. With the causal mask on, the upper-right triangle is empty: The sees only itself, sat sees The, cat and sat, and mat sees all six tokens. A highlighted band follows the query row from The to mat as you step. The right-hand column shows the printed row total, which reads 1.000 on every row, in both mask states. Hovering or focusing a cell reports the query, the key and the three-decimal weight, or says the cell was masked before softmax and has no weight. A checkbox turns the mask off, and the empty triangle fills with weights.
Trade-offs
| Variant | Good for | Cost | Fails when |
|---|---|---|---|
| Full self-attention | Quality baseline, contexts within the trained range | O(n²) time and score memory | Context is long or concurrency is high |
| Sliding-window attention | Local structure, streaming, cheap long inputs | Cannot reach distant evidence directly | The answer depends on a fact far from the question |
| Sparse / linear approximations | Very long documents at scale | Quality loss, extra implementation complexity | You need exact recall over the whole context |
| Flash attention kernels | Same math with less memory traffic | Kernel and hardware constraints | Your dtype, head dimension or device is unsupported |
| Cross-attention | Conditioning on a separate context, as in encoder-decoder | Two sources to manage, separate masks | You assume it is interchangeable with self-attention |
Failure modes
The mask is a no-op. A mask with the wrong shape broadcasts silently, or is applied after the softmax, and the model reads the future during training. Debugging step: change one token after position and assert the logits at are bit-identical.
Padding is attended as content. Batches contain pad tokens that other positions weight as if they were text. Debugging step: score the same example alone and inside a padded batch; a difference means padding was not masked.
Attention saturates. Logits are large, softmax collapses onto one position, and gradients vanish. Debugging step: log the entropy of the attention distribution per head; a sustained near-zero entropy means the scaling or the initialisation is wrong.
Weights are read as explanations. A high weight is taken as proof of causal influence. Debugging step: ablate or perturb the attended token and measure the output change; weights and causal effect frequently disagree.
In production
Prefill computes the full score matrix in parallel; a 32k-token prompt is 1024 times more pairwise work than 1k tokens. That is the arithmetic behind long-context latency and price curves.
At decode time the picture changes: one query against a long cached key/value set, so attention becomes memory-bandwidth-bound rather than compute-bound. This is why optimisations that reduce cache size (grouped-query attention, quantised caches) can help decode more than faster matrix maths.
Quality does not improve monotonically with context. Evidence placed in the middle of a long prompt is used less reliably than evidence at the edges, so retrieve-then-place ordering is still an engineering decision even when the window is large enough.
Common misconceptions
People think: Attention weights show why the model produced an answer.
Actually: Weights show where one head read from at one layer. Ablation studies find many attention patterns can be changed with little effect on output, so weights are evidence, not explanation.
People think: Self-attention is doing retrieval over the prompt the way a vector index does.
Actually: It is a learned, per-layer, multi-head weighting with no index and no exact-match guarantee. The retrieval analogy is useful for intuition, not for predicting recall.
A third: that "attention is all you need" means the feed-forward blocks are incidental. They hold most of the parameters and do most of the per-position transformation.
Interview angle
Self-attention questions are shape questions in disguise. Expect to derive the dimensions, explain the scaling, describe what a causal mask does, and reason about why cost is quadratic. The discriminating follow-up is usually about long context: what breaks first, and which part of the cost a cache or a kernel actually reduces. See Q/K/V and multi-head attention, plus the questions at /interview.
Sources
- Vaswani et al., Attention Is All You Need — scaled dot-product attention and its complexity: https://arxiv.org/abs/1706.03762
- Harvard NLP, The Annotated Transformer — executable reference implementation: https://nlp.seas.harvard.edu/annotated-transformer/
- Jain & Wallace, Attention is not Explanation — evidence that attention weights and causal importance diverge: https://arxiv.org/abs/1902.10186
- Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — why the same math can be much faster: https://arxiv.org/abs/2205.14135