Why it matters
A language model has no notion of a sentence as an output unit. It computes a probability distribution for one position, and the application decides what to do with it, feeds the result back and repeats until a stopping condition. Latency that scales with answer length, the impossibility of editing a half-written answer, and the need for stop sequences and token caps all follow from that loop rather than from model size.
Mental model
It is writing a sentence one letter at a time with no eraser. Each letter is chosen given everything already written. The image breaks at the start: the model does not plan the sentence and then transcribe it, so there is no draft to correct, only a prefix that keeps growing.
How it works
- 1Tokenise the full prompt and run a prefill pass that computes keys and values for every prompt position in one parallel operation.
- 2Read the distribution at the final position, apply the decoding rule, and select one token id.
- 3Append that token and run a forward pass that reuses the cached keys and values instead of recomputing the prefix.
- 4Stop when the model emits an end-of-sequence token, matches a configured stop string, or reaches the output token cap.
- 5Detokenise the accumulated ids once at the end, since tokens do not align with word or character boundaries.
- 6Record prompt tokens, completion tokens and finish reason per request, because those three fields explain most cost and truncation incidents.
Step 1 of 6
Tokenise the full prompt and run a prefill pass that computes keys and values for every prompt position in one parallel operation.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Tokenise the full prompt and run a prefill pass that computes keys and values for every p… 2. Read the distribution at the final position, apply the decoding rule, and select one toke… 3. Append that token and run a forward pass that reuses the cached keys and values instead o… 4. Stop when the model emits an end-of-sequence token, matches a configured stop string, or … 5. Detokenise the accumulated ids once at the end, since tokens do not align with word or ch… 6. Record prompt tokens, completion tokens and finish reason per request, because those thre…
- Tokenise the full prompt and run a prefill pass that computes keys and values for every prompt position in one parallel operation.
- Read the distribution at the final position, apply the decoding rule, and select one token id.
- Append that token and run a forward pass that reuses the cached keys and values instead of recomputing the prefix.
- Stop when the model emits an end-of-sequence token, matches a configured stop string, or reaches the output token cap.
- Detokenise the accumulated ids once at the end, since tokens do not align with word or character boundaries.
- Record prompt tokens, completion tokens and finish reason per request, because those three fields explain most cost and truncation incidents.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Single-pass answer | Low latency, simple client | No chance to check intermediate work | The task needs verification between steps |
| Streamed token loop | Early feedback, responsive feel | Partial output must be handled and sometimes discarded | Downstream parsing needs the complete document |
| Long outputs | Detailed answers, worked reasoning | Decode time and cost grow with every token | The budget is interactive latency |
| Reuse the KV cache | Cheap continuation of a prefix | Memory held per active sequence | Concurrency is high and sequences are long |
Good for
Low latency, simple client
Cost
No chance to check intermediate work
Fails when
The task needs verification between steps
- The task needs verification between steps
- Downstream parsing needs the complete document
- The budget is interactive latency
- Concurrency is high and sequences are long
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Single-pass answer is good for Low latency, simple client, costs No chance to check intermediate work, and fails when The task needs verification between steps. Streamed token loop is good for Early feedback, responsive feel, costs Partial output must be handled and sometimes discarded, and fails when Downstream parsing needs the complete document. Long outputs is good for Detailed answers, worked reasoning, costs Decode time and cost grow with every token, and fails when The budget is interactive latency. Reuse the KV cache is good for Cheap continuation of a prefix, costs Memory held per active sequence, and fails when Concurrency is high and sequences are long
Failure modes
- Prefix drift. A wrong early token makes the rest plausible but wrong, and the model cannot go back. Debug: compare the answer with a run that constrains the first sentence, and check whether the error is already present in the first few tokens.
- Truncated finish. The response stops at the token cap mid-sentence and the client treats it as complete. Debug: log the finish reason and assert it is a natural stop before parsing.
- Recomputed prefix. The server reprocesses the whole prompt on every call because caching is off, so latency scales with prompt length. Debug: compare time to first token for a short and a long prompt with the same output length.
In production
Watch time to first token and tokens per second separately, cap output tokens per route, and alert on the share of responses that end by length rather than by a stop condition.
Common misconceptions
People think: The model writes the whole answer and then returns it.
Actually: It commits to each token before seeing the next, so the answer is built incrementally and cannot be revised.
People think: A bigger model removes the need for output limits.
Actually: Every extra token is another forward pass; limits bound latency and cost regardless of model size.
Interview angle
Explain the loop, then connect it to observable behaviour: why latency grows with output length, why an early error is unrecoverable, and why prefill and decode respond to different fixes. A weak answer repeats next-token prediction as a slogan and cannot say what happens between two adjacent tokens.
Sources
- Vaswani et al., Attention Is All You Need — the transformer architecture that generation loops run on.
- Brown et al., Language Models are Few-Shot Learners — autoregressive next-token training and in-context behaviour.
- Pope et al., Efficiently Scaling Transformer Inference — prefill versus decode cost and KV cache memory.