Why it matters
A model often needs to condition on something it is not generating: a source sentence, an image, a retrieved document. Self-attention cannot express that because it mixes one sequence with itself. Cross-attention provides a learned interface between two representations, so the decoder can consult the context at every layer without the context having to be flattened into a single vector or repeated in the prompt.
Mental model
Cross-attention is a translator glancing at the source text while writing: the sentence being written supplies the questions, the source supplies the answers. The metaphor breaks because the glance is not sequential — every output position can read every source position at once, with weights learned rather than following word order.
How it works
Hover or focus a cell to read its exact value and printed row total. Dashed cells are removed by the padding mask before normalisation.
- query length
- 4
- context length
- 5
- score shape
- 4 × 5, not square
Step 1 of 4
Two lengths, one rectangle
Four queries and five source slots give a 4 × 5 score grid. Self-attention would be square here; cross-attention is rectangular whenever the two sequences differ in length.
Next: Row sat
Figure. Four decoder positions querying five source positions, illustrative weights. Rows are the decoder's queries and columns the encoder's keys, so the grid is rectangular: its shape follows the two sequence lengths, not one length. A cell is the weight a decoder position puts on a source slot, and with context padding masked before normalisation every printed row totals 1.000.
What to notice. The mask removes the pad column and the row renormalises over what remains: sat's 0.08, 0.12, 0.45 and 0.20 become 0.094, 0.141, 0.530 and 0.235. Turn the mask off and the pad cell takes its share while the row still totals 1.000, so a row total cannot tell you whether padding was masked. This is one head of one layer; a real layer runs one rectangular score grid per head, and keys and values are projected from the encoder output once per request and cached.
Text version
A four by five matrix. Rows are the decoder positions The, cat, sat and on; columns are the source positions Le, chat, s'est, assis and pad. The context-padding mask is on by default, so the pad column is dashed and empty in every row. With the mask on, The prints 0.556, 0.222, 0.111 and 0.111; cat prints 0.111, 0.611, 0.167 and 0.111; sat prints 0.094, 0.141, 0.530 and 0.235; on prints 0.118, 0.118, 0.176 and 0.588. Every row-total cell reads 1.000. Steps show the whole rectangle, the sat row, the pad column across all four rows, and the mask off, where the pad cells fill and the rows still total 1.000. Hovering a cell reports the decoder position, the source position and the printed weight.
- Produce the context representation once, for example by running an encoder over the source sequence.
- Project the decoder's current state into queries using the decoder's own weight matrix.
- Project the context into keys and values using cross-attention weight matrices, which are separate from the encoder's self-attention weights.
- Compute rectangular scores of shape output length by context length, scale and mask context padding.
- Softmax over context positions and mix the context values into the decoder state.
- Cache the projected keys and values per request so they are not recomputed at every decoding step.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Cross-attention to an encoder | Rich conditioning, full bidirectional source view | Extra parameters, two caches to manage | The source is short and could simply be concatenated |
| Prompt concatenation | No architecture change, works with any decoder | Context consumes window and is re-read every step | The source is long or reused across many queries |
| Per-layer cross-attention | Deep interaction between streams | Compute grows with layers times context length | Latency budget is tight |
| Adapter cross-attention | Injects external memory without retraining the base | New parameters and a second training loop | The base model's layers are not accessible |
Good for
Rich conditioning, full bidirectional source view
Cost
Extra parameters, two caches to manage
Fails when
The source is short and could simply be concatenated
- The source is short and could simply be concatenated
- The source is long or reused across many queries
- Latency budget is tight
- The base model's layers are not accessible
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Cross-attention to an encoder is good for Rich conditioning, full bidirectional source view, costs Extra parameters, two caches to manage, and fails when The source is short and could simply be concatenated. Prompt concatenation is good for No architecture change, works with any decoder, costs Context consumes window and is re-read every step, and fails when The source is long or reused across many queries. Per-layer cross-attention is good for Deep interaction between streams, costs Compute grows with layers times context length, and fails when Latency budget is tight. Adapter cross-attention is good for Injects external memory without retraining the base, costs New parameters and a second training loop, and fails when The base model's layers are not accessible
Failure modes
- Context padding attended. The decoder reads pad positions as if they were content and output quality drops with batch size. Debug: compare a single-example decode against the same example in a padded batch.
- Keys and values recomputed per step. Decode latency scales with context length times steps. Debug: profile to confirm the context projection runs once per request.
- Wrong stream supplies queries. Swapping query and context projections still runs but conditions the source on the output. Debug: check the shape of the query tensor against the output length, not the context length.
In production
Cache context keys and values per request and monitor memory separately from the self-attention cache. Track output quality against context length, since attention dilution over a long source is a quiet failure that aggregate metrics hide.
Common misconceptions
People think: Cross-attention and self-attention are different mechanisms.
Actually: They are the same operation with different inputs; only the source of keys and values changes, which is why one implementation serves both.
People think: Concatenating context into the prompt is equivalent.
Actually: Concatenation makes the context part of the causal stream and pays for it at every step; cross-attention reads a fixed representation and can be cached.
Interview angle
A strong answer states the only structural difference from self-attention, gives the rectangular shape, and explains caching and masking for the context stream. It should compare with prompt concatenation on cost. Weak answers describe it as 'attention to another sequence' without saying which tensor supplies queries or why the score matrix is not square.
Sources
- Vaswani et al., Attention Is All You Need — encoder-decoder cross-attention in the transformer.
- Bahdanau, Cho & Bengio, Neural Machine Translation by Jointly Learning to Align and Translate — the original soft-alignment formulation.
- Alammar, The Illustrated Transformer — a visual walkthrough of the two attention paths.