Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 15 of 33

Encoder/decoder

0 of 33 complete

Why it matters

Some tasks have an input that should be understood in full before any output is produced. A decoder-only model can only condition on the input through the same causal stream, which works but wastes the ability to look ahead within the source. The encoder-decoder split lets the encoder build a bidirectional representation once, and the decoder reuse it at every generation step, which is why translation and summarisation architectures kept the shape for years.

Mental model

It is a translator with a research phase and a writing phase: first read and annotate the whole source, then write the target sentence while glancing back at the annotations. The metaphor breaks because the annotations are not notes in a shared language — they are vectors that only the decoder's learned cross-attention can interpret.

How it works

Two stacks and the cross-attention bridgeInteractive
signalcrossloop

Step 1 of 6

Encode the source

The encoder sees the whole source at once with bidirectional attention, so each position can read every other source position.

Active layers

  • Source tokens
  • Encoder · N layers
  • Self-attention

Hover a layer to read what it holds.

Next: Produce encoder states

Encoder
1 pass per request
Decoder
1 pass per token
Bridge
cross-attention · K, V
Step 1/6 · Encode the source

Figure. A source sequence is encoded bidirectionally once, then a causal decoder generates output while reading the encoder's states through cross-attention at every layer. A right-hand loop returns the sampled token to the top of the decoder group; the decoder's own queries come from its current state.

What to notice. The encoder runs once per request and its states are cached; the loop shows the decoder running again per token. Cross-attention sits inside each decoder layer, so the vertical order inside the decoder group is masked self-attention, then cross-attention, then feed-forward. N layers and the single loop are illustrative.

Text version

A vertical stack with two groups. The encoder group holds bidirectional self-attention and a feed-forward block. The decoder group holds masked self-attention, cross-attention and a feed-forward block, ending in vocabulary logits. A dashed edge from the encoder output to the decoder's cross-attention carries keys and values, and a normal edge from the decoder self-attention carries queries. A dash-dot loop runs from the logits back to the decoder group, labelled next token. Six steps walk encoding once, caching the states, causal decoding, cross-attending, predicting and looping, and why the split is paid once versus once per token.

  1. Encode the source sequence with bidirectional self-attention, producing one state per input position.
  2. Initialise the decoder with a start token and run causal self-attention over the output produced so far.
  3. At each decoder layer, run cross-attention where queries come from the decoder and keys and values come from the encoder output.
  4. Project the final decoder state to vocabulary logits and predict the next output token.
  5. Train with teacher forcing, feeding the gold prefix while masking future output positions.
  6. At inference, cache encoder states once and reuse them across all decoding steps.

Trade-offs

ChoiceGood forCostFails when
Encoder-decoderClean separation, bidirectional source encodingTwo stacks, two caches, more parametersThe task is open-ended chat with no distinct source
Decoder-onlyOne stack, simple scaling, flexible promptsSource is processed causally, no bidirectional passInput understanding needs full-document context
Shared encoder-decoder weightsFewer parameters, regularisationLess specialisation per roleSource and target distributions differ a lot
Long encoder inputWhole document available to the decoderEncoder attention cost grows quadraticallyInputs exceed the trained length
Trade-off mapInteractive

Good for

Clean separation, bidirectional source encoding

Cost

Two stacks, two caches, more parameters

Fails when

The task is open-ended chat with no distinct source

  1. The task is open-ended chat with no distinct source
  2. Input understanding needs full-document context
  3. Source and target distributions differ a lot
  4. Inputs exceed the trained length

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Encoder-decoder is good for Clean separation, bidirectional source encoding, costs Two stacks, two caches, more parameters, and fails when The task is open-ended chat with no distinct source. Decoder-only is good for One stack, simple scaling, flexible prompts, costs Source is processed causally, no bidirectional pass, and fails when Input understanding needs full-document context. Shared encoder-decoder weights is good for Fewer parameters, regularisation, costs Less specialisation per role, and fails when Source and target distributions differ a lot. Long encoder input is good for Whole document available to the decoder, costs Encoder attention cost grows quadratically, and fails when Inputs exceed the trained length

Failure modes

  • Cross-attention mask inverted. The decoder attends to padding positions in the encoder output and emits filler. Debug: visualise the cross-attention weights for one batch and check that pad positions receive zero.
  • Encoder states recomputed per step. Decoding is far slower than expected because the source is re-encoded for every token. Debug: profile one generation and confirm the encoder runs once per request, not once per step.
  • Teacher-forcing exposure gap. The model only sees gold prefixes in training and drifts once its own errors enter the context. Debug: evaluate with scheduled sampling or free-running decoding and compare to teacher-forced loss.

In production

Cache encoder output per request and monitor decoder step latency separately from encoding latency, since they scale with different lengths. Track output length distribution against the training data; a decoder that has never seen long targets tends to stop early.

Common misconceptions

People think: Encoder-decoder is obsolete now that decoder-only models scale.

Actually: The split still wins where the input is a distinct document and latency allows one encoding pass; the choice is about task shape and serving cost, not recency.

People think: The encoder compresses the input into one vector.

Actually: It produces a state per position and the decoder attends over all of them; a single bottleneck vector was the pre-attention design.

Interview angle

A strong answer draws the two stacks, explains which tensors cross the boundary and why cross-attention uses decoder queries, and names the tasks where the split pays. It should compare with decoder-only on simplicity and serving cost. Weak answers describe it as 'seq2seq' without saying what the decoder attends to or why the encoder is bidirectional.

Sources