Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 20 of 33

Cross-attention

0 of 33 complete

Why it matters

A model often needs to condition on something it is not generating: a source sentence, an image, a retrieved document. Self-attention cannot express that because it mixes one sequence with itself. Cross-attention provides a learned interface between two representations, so the decoder can consult the context at every layer without the context having to be flattened into a single vector or repeated in the prompt.

Mental model

Cross-attention is a translator glancing at the source text while writing: the sentence being written supplies the questions, the source supplies the answers. The metaphor breaks because the glance is not sequential — every output position can read every source position at once, with weights learned rather than following word order.

How it works

Two sequences · four queries over five source slotsInteractive
00.700padding
decoder query ↓source key →
Lechats'estassispadΣ
The1.000
cat1.000
sat1.000
on1.000

Hover or focus a cell to read its exact value and printed row total. Dashed cells are removed by the padding mask before normalisation.

query length
4
context length
5
score shape
4 × 5, not square

Step 1 of 4

Two lengths, one rectangle

Four queries and five source slots give a 4 × 5 score grid. Self-attention would be square here; cross-attention is rectangular whenever the two sequences differ in length.

Next: Row sat

Step 1/4 · Two lengths, one rectangle

Figure. Four decoder positions querying five source positions, illustrative weights. Rows are the decoder's queries and columns the encoder's keys, so the grid is rectangular: its shape follows the two sequence lengths, not one length. A cell is the weight a decoder position puts on a source slot, and with context padding masked before normalisation every printed row totals 1.000.

What to notice. The mask removes the pad column and the row renormalises over what remains: sat's 0.08, 0.12, 0.45 and 0.20 become 0.094, 0.141, 0.530 and 0.235. Turn the mask off and the pad cell takes its share while the row still totals 1.000, so a row total cannot tell you whether padding was masked. This is one head of one layer; a real layer runs one rectangular score grid per head, and keys and values are projected from the encoder output once per request and cached.

Text version

A four by five matrix. Rows are the decoder positions The, cat, sat and on; columns are the source positions Le, chat, s'est, assis and pad. The context-padding mask is on by default, so the pad column is dashed and empty in every row. With the mask on, The prints 0.556, 0.222, 0.111 and 0.111; cat prints 0.111, 0.611, 0.167 and 0.111; sat prints 0.094, 0.141, 0.530 and 0.235; on prints 0.118, 0.118, 0.176 and 0.588. Every row-total cell reads 1.000. Steps show the whole rectangle, the sat row, the pad column across all four rows, and the mask off, where the pad cells fill and the rows still total 1.000. Hovering a cell reports the decoder position, the source position and the printed weight.

  1. Produce the context representation once, for example by running an encoder over the source sequence.
  2. Project the decoder's current state into queries using the decoder's own weight matrix.
  3. Project the context into keys and values using cross-attention weight matrices, which are separate from the encoder's self-attention weights.
  4. Compute rectangular scores of shape output length by context length, scale and mask context padding.
  5. Softmax over context positions and mix the context values into the decoder state.
  6. Cache the projected keys and values per request so they are not recomputed at every decoding step.

Trade-offs

ChoiceGood forCostFails when
Cross-attention to an encoderRich conditioning, full bidirectional source viewExtra parameters, two caches to manageThe source is short and could simply be concatenated
Prompt concatenationNo architecture change, works with any decoderContext consumes window and is re-read every stepThe source is long or reused across many queries
Per-layer cross-attentionDeep interaction between streamsCompute grows with layers times context lengthLatency budget is tight
Adapter cross-attentionInjects external memory without retraining the baseNew parameters and a second training loopThe base model's layers are not accessible
Trade-off mapInteractive

Good for

Rich conditioning, full bidirectional source view

Cost

Extra parameters, two caches to manage

Fails when

The source is short and could simply be concatenated

  1. The source is short and could simply be concatenated
  2. The source is long or reused across many queries
  3. Latency budget is tight
  4. The base model's layers are not accessible

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Cross-attention to an encoder is good for Rich conditioning, full bidirectional source view, costs Extra parameters, two caches to manage, and fails when The source is short and could simply be concatenated. Prompt concatenation is good for No architecture change, works with any decoder, costs Context consumes window and is re-read every step, and fails when The source is long or reused across many queries. Per-layer cross-attention is good for Deep interaction between streams, costs Compute grows with layers times context length, and fails when Latency budget is tight. Adapter cross-attention is good for Injects external memory without retraining the base, costs New parameters and a second training loop, and fails when The base model's layers are not accessible

Failure modes

  • Context padding attended. The decoder reads pad positions as if they were content and output quality drops with batch size. Debug: compare a single-example decode against the same example in a padded batch.
  • Keys and values recomputed per step. Decode latency scales with context length times steps. Debug: profile to confirm the context projection runs once per request.
  • Wrong stream supplies queries. Swapping query and context projections still runs but conditions the source on the output. Debug: check the shape of the query tensor against the output length, not the context length.

In production

Cache context keys and values per request and monitor memory separately from the self-attention cache. Track output quality against context length, since attention dilution over a long source is a quiet failure that aggregate metrics hide.

Common misconceptions

People think: Cross-attention and self-attention are different mechanisms.

Actually: They are the same operation with different inputs; only the source of keys and values changes, which is why one implementation serves both.

People think: Concatenating context into the prompt is equivalent.

Actually: Concatenation makes the context part of the causal stream and pays for it at every step; cross-attention reads a fixed representation and can be cached.

Interview angle

A strong answer states the only structural difference from self-attention, gives the rectangular shape, and explains caching and masking for the context stream. It should compare with prompt concatenation on cost. Weak answers describe it as 'attention to another sequence' without saying which tensor supplies queries or why the score matrix is not square.

Sources