Why it matters
Some tasks have an input that should be understood in full before any output is produced. A decoder-only model can only condition on the input through the same causal stream, which works but wastes the ability to look ahead within the source. The encoder-decoder split lets the encoder build a bidirectional representation once, and the decoder reuse it at every generation step, which is why translation and summarisation architectures kept the shape for years.
Mental model
It is a translator with a research phase and a writing phase: first read and annotate the whole source, then write the target sentence while glancing back at the annotations. The metaphor breaks because the annotations are not notes in a shared language — they are vectors that only the decoder's learned cross-attention can interpret.
How it works
Step 1 of 6
Encode the source
The encoder sees the whole source at once with bidirectional attention, so each position can read every other source position.
Active layers
- Source tokens
- Encoder · N layers
- Self-attention
Hover a layer to read what it holds.
Next: Produce encoder states
- Encoder
- 1 pass per request
- Decoder
- 1 pass per token
- Bridge
- cross-attention · K, V
Figure. A source sequence is encoded bidirectionally once, then a causal decoder generates output while reading the encoder's states through cross-attention at every layer. A right-hand loop returns the sampled token to the top of the decoder group; the decoder's own queries come from its current state.
What to notice. The encoder runs once per request and its states are cached; the loop shows the decoder running again per token. Cross-attention sits inside each decoder layer, so the vertical order inside the decoder group is masked self-attention, then cross-attention, then feed-forward. N layers and the single loop are illustrative.
Text version
A vertical stack with two groups. The encoder group holds bidirectional self-attention and a feed-forward block. The decoder group holds masked self-attention, cross-attention and a feed-forward block, ending in vocabulary logits. A dashed edge from the encoder output to the decoder's cross-attention carries keys and values, and a normal edge from the decoder self-attention carries queries. A dash-dot loop runs from the logits back to the decoder group, labelled next token. Six steps walk encoding once, caching the states, causal decoding, cross-attending, predicting and looping, and why the split is paid once versus once per token.
- Encode the source sequence with bidirectional self-attention, producing one state per input position.
- Initialise the decoder with a start token and run causal self-attention over the output produced so far.
- At each decoder layer, run cross-attention where queries come from the decoder and keys and values come from the encoder output.
- Project the final decoder state to vocabulary logits and predict the next output token.
- Train with teacher forcing, feeding the gold prefix while masking future output positions.
- At inference, cache encoder states once and reuse them across all decoding steps.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Encoder-decoder | Clean separation, bidirectional source encoding | Two stacks, two caches, more parameters | The task is open-ended chat with no distinct source |
| Decoder-only | One stack, simple scaling, flexible prompts | Source is processed causally, no bidirectional pass | Input understanding needs full-document context |
| Shared encoder-decoder weights | Fewer parameters, regularisation | Less specialisation per role | Source and target distributions differ a lot |
| Long encoder input | Whole document available to the decoder | Encoder attention cost grows quadratically | Inputs exceed the trained length |
Good for
Clean separation, bidirectional source encoding
Cost
Two stacks, two caches, more parameters
Fails when
The task is open-ended chat with no distinct source
- The task is open-ended chat with no distinct source
- Input understanding needs full-document context
- Source and target distributions differ a lot
- Inputs exceed the trained length
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Encoder-decoder is good for Clean separation, bidirectional source encoding, costs Two stacks, two caches, more parameters, and fails when The task is open-ended chat with no distinct source. Decoder-only is good for One stack, simple scaling, flexible prompts, costs Source is processed causally, no bidirectional pass, and fails when Input understanding needs full-document context. Shared encoder-decoder weights is good for Fewer parameters, regularisation, costs Less specialisation per role, and fails when Source and target distributions differ a lot. Long encoder input is good for Whole document available to the decoder, costs Encoder attention cost grows quadratically, and fails when Inputs exceed the trained length
Failure modes
- Cross-attention mask inverted. The decoder attends to padding positions in the encoder output and emits filler. Debug: visualise the cross-attention weights for one batch and check that pad positions receive zero.
- Encoder states recomputed per step. Decoding is far slower than expected because the source is re-encoded for every token. Debug: profile one generation and confirm the encoder runs once per request, not once per step.
- Teacher-forcing exposure gap. The model only sees gold prefixes in training and drifts once its own errors enter the context. Debug: evaluate with scheduled sampling or free-running decoding and compare to teacher-forced loss.
In production
Cache encoder output per request and monitor decoder step latency separately from encoding latency, since they scale with different lengths. Track output length distribution against the training data; a decoder that has never seen long targets tends to stop early.
Common misconceptions
People think: Encoder-decoder is obsolete now that decoder-only models scale.
Actually: The split still wins where the input is a distinct document and latency allows one encoding pass; the choice is about task shape and serving cost, not recency.
People think: The encoder compresses the input into one vector.
Actually: It produces a state per position and the decoder attends over all of them; a single bottleneck vector was the pre-attention design.
Interview angle
A strong answer draws the two stacks, explains which tensors cross the boundary and why cross-attention uses decoder queries, and names the tasks where the split pays. It should compare with decoder-only on simplicity and serving cost. Weak answers describe it as 'seq2seq' without saying what the decoder attends to or why the encoder is bidirectional.
Sources
- Vaswani et al., Attention Is All You Need — the encoder-decoder transformer and cross-attention.
- Bahdanau, Cho & Bengio, Neural Machine Translation by Jointly Learning to Align and Translate — the alignment mechanism that became attention.
- Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — a consistent encoder-decoder recipe across tasks.