Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 16 of 33

Self-attention

0 of 33 complete

Why it matters

Self-attention is the mechanism that makes a language model context-sensitive. The same token gets a different representation depending on what surrounds it: "bank" near "river" and "bank" near "loan" end up in different places in vector space, because each position reads from its neighbours and reweights itself accordingly.

It is also where the cost lives. Attention compares every position with every other position, so doubling the context quadruples the pairwise work and the score matrix. Long-context pricing, the KV cache, grouped-query attention and flash kernels are all responses to that one quadratic term.

Understanding the mechanism at the level of shapes and masks is what lets you debug the failures: future-token leakage, padding that is attended as content, or a context that is technically accepted but effectively ignored.

Mental model: a soft dictionary lookup

Each position asks a question (its query), every position advertises what it contains (its key), and every position carries content (its value). The query is compared with every key, the scores are normalised into weights, and the position receives a weighted blend of values.

Retrieval engineers recognise this immediately: it is a search over the sequence, with the retrieval weights learned rather than computed by cosine similarity.

The analogy breaks in three ways. The lookup is soft — every key gets some weight, not just the best match. All positions do this simultaneously, in parallel, rather than one query at a time. And the keys and values are not authored descriptions; they are linear projections learned end to end, so what a key "advertises" is whatever helped the training loss.

How it works

  1. Project the input. From the block's input XX, compute queries, keys and values with learned weight matrices: Q=XWQQ = XW_Q, K=XWKK = XW_K, V=XWVV = XW_V.
  2. Score every pair. Multiply QQ by K⊤K^\top to get an n×nn \times n matrix of raw affinities for a sequence of length nn.
  3. Scale. Divide by dk\sqrt{d_k}, the square root of the key dimension. Without it, dot products grow with dimension and push softmax into saturation.
  4. Mask. Add a causal mask for decoder self-attention so position tt cannot read positions after tt. Mask padding so pad tokens are not treated as content.
  5. Normalise. Apply softmax along the key axis, so each row of weights sums to 1.
  6. Blend the values. Multiply the weights by VV. Each output row is a weighted sum of value vectors from the positions it attended to.
  7. Repeat per head and project. Multiple heads run in parallel with their own projections; their outputs are concatenated and mixed by an output projection (see multi-head attention).
  8. Stack. The result is added to the residual stream and passed to the next block.

The math

Attention(Q,K,V)=softmax ⁣(QK⊤dk+M)V,\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)V,

where MM is the additive mask: 00 for allowed positions and −∞-\infty for forbidden ones. For a sequence of length nn and head dimension dkd_k:

time=O(n2dk),score memory=O(n2) per head.\text{time} = O(n^2 d_k), \qquad \text{score memory} = O(n^2) \text{ per head}.

The quadratic term is in nn, not in the parameter count. That distinction explains why a model with fixed weights becomes progressively more expensive per token as the prompt grows.

A minimal implementation

import numpy as np
 
def softmax(x: np.ndarray, axis: int = -1) -> np.ndarray:
    x = x - x.max(axis=axis, keepdims=True)
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)
 
def self_attention(Q, K, V, causal: bool = True):
    d_k = Q.shape[-1]
    scores = Q @ np.swapaxes(K, -1, -2) / np.sqrt(d_k)   # (..., n, n)
    if causal:
        n = scores.shape[-1]
        future = np.triu(np.ones((n, n), dtype=bool), k=1)
        scores = np.where(future, -np.inf, scores)
    weights = softmax(scores, axis=-1)                    # rows sum to 1
    return weights @ V, weights
 
rng = np.random.default_rng(0)
X = rng.normal(size=(1, 4, 8))                            # batch, tokens, d_model
Wq, Wk, Wv = (rng.normal(size=(8, 8)) for _ in range(3))
out, w = self_attention(X @ Wq, X @ Wk, X @ Wv)
 
print(out.shape, w.shape)          # (1, 4, 8) (1, 4, 4)
print(w[0, -1].round(2))           # last token may read all four positions

One trap hides in the mask. If a row is fully masked, max returns -inf, subtraction yields nan, and the whole row becomes nan instead of zeros. Causal masks avoid this because the first position always attends to itself; padding masks with an empty sequence do not.

Attention matrix · one head, six tokensInteractive

weightmasked

query rows ↓keys → columns
ThecatsatonthematΣ
The1.0001 key
cat1.0002 keys
sat1.0003 keys
on1.0004 keys
the1.0005 keys
mat1.0006 keys

Row 1 · query “The” reads from itself only.

Step 1/6 · query: The

Illustrative single-head weights, renormalised after masking. Real models run several heads per layer. Toggle the mask off to see what the same head would attend to without it — the future cells fill in, and the causal property disappears.

Figure. One head over six tokens, with illustrative weights. Each row is a query and each column a key; a cell shows the weight that query places on that key, renormalised so the row sums to 1.000. Cells are printed to three decimals with the rounding residue assigned by largest remainder, so the printed row also adds to exactly 1.000.

What to notice. The mask is applied before softmax, not after. Future positions are set to −∞, so they contribute exactly zero once the surviving weights are renormalised; masking after normalisation would leave the row summing to less than 1. The Σ column adds the printed cells, not the unrounded ones.

Source: Vaswani et al., Attention Is All You Need (2017), arXiv:1706.03762. Weights are illustrative, not read from a trained model.

Text version

A six by six attention matrix for the tokens The, cat, sat, on, the and mat. Rows are queries, columns are keys, and each filled cell prints its weight to three decimals and shades it darker for a larger weight. With the causal mask on, the upper-right triangle is empty: The sees only itself, sat sees The, cat and sat, and mat sees all six tokens. A highlighted band follows the query row from The to mat as you step. The right-hand column shows the printed row total, which reads 1.000 on every row, in both mask states. Hovering or focusing a cell reports the query, the key and the three-decimal weight, or says the cell was masked before softmax and has no weight. A checkbox turns the mask off, and the empty triangle fills with weights.

Trade-offs

VariantGood forCostFails when
Full self-attentionQuality baseline, contexts within the trained rangeO(n²) time and score memoryContext is long or concurrency is high
Sliding-window attentionLocal structure, streaming, cheap long inputsCannot reach distant evidence directlyThe answer depends on a fact far from the question
Sparse / linear approximationsVery long documents at scaleQuality loss, extra implementation complexityYou need exact recall over the whole context
Flash attention kernelsSame math with less memory trafficKernel and hardware constraintsYour dtype, head dimension or device is unsupported
Cross-attentionConditioning on a separate context, as in encoder-decoderTwo sources to manage, separate masksYou assume it is interchangeable with self-attention

Failure modes

The mask is a no-op. A mask with the wrong shape broadcasts silently, or is applied after the softmax, and the model reads the future during training. Debugging step: change one token after position tt and assert the logits at tt are bit-identical.

Padding is attended as content. Batches contain pad tokens that other positions weight as if they were text. Debugging step: score the same example alone and inside a padded batch; a difference means padding was not masked.

Attention saturates. Logits are large, softmax collapses onto one position, and gradients vanish. Debugging step: log the entropy of the attention distribution per head; a sustained near-zero entropy means the scaling or the initialisation is wrong.

Weights are read as explanations. A high weight is taken as proof of causal influence. Debugging step: ablate or perturb the attended token and measure the output change; weights and causal effect frequently disagree.

In production

Prefill computes the full n×nn \times n score matrix in parallel; a 32k-token prompt is 1024 times more pairwise work than 1k tokens. That is the arithmetic behind long-context latency and price curves.

At decode time the picture changes: one query against a long cached key/value set, so attention becomes memory-bandwidth-bound rather than compute-bound. This is why optimisations that reduce cache size (grouped-query attention, quantised caches) can help decode more than faster matrix maths.

Quality does not improve monotonically with context. Evidence placed in the middle of a long prompt is used less reliably than evidence at the edges, so retrieve-then-place ordering is still an engineering decision even when the window is large enough.

Common misconceptions

People think: Attention weights show why the model produced an answer.

Actually: Weights show where one head read from at one layer. Ablation studies find many attention patterns can be changed with little effect on output, so weights are evidence, not explanation.

People think: Self-attention is doing retrieval over the prompt the way a vector index does.

Actually: It is a learned, per-layer, multi-head weighting with no index and no exact-match guarantee. The retrieval analogy is useful for intuition, not for predicting recall.

A third: that "attention is all you need" means the feed-forward blocks are incidental. They hold most of the parameters and do most of the per-position transformation.

Interview angle

Self-attention questions are shape questions in disguise. Expect to derive the QK⊤QK^\top dimensions, explain the dk\sqrt{d_k} scaling, describe what a causal mask does, and reason about why cost is quadratic. The discriminating follow-up is usually about long context: what breaks first, and which part of the cost a cache or a kernel actually reduces. See Q/K/V and multi-head attention, plus the questions at /interview.

Sources