Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 12 of 28

Autoregressive generation

0 of 28 complete

Why it matters

A language model has no notion of a sentence as an output unit. It computes a probability distribution for one position, and the application decides what to do with it, feeds the result back and repeats until a stopping condition. Latency that scales with answer length, the impossibility of editing a half-written answer, and the need for stop sequences and token caps all follow from that loop rather than from model size.

Mental model

It is writing a sentence one letter at a time with no eraser. Each letter is chosen given everything already written. The image breaks at the start: the model does not plan the sentence and then transcribe it, so there is no draft to correct, only a prefix that keeps growing.

How it works

Mechanism, step by stepInteractive
  1. 1Tokenise the full prompt and run a prefill pass that computes keys and values for every prompt position in one parallel operation.
  2. 2Read the distribution at the final position, apply the decoding rule, and select one token id.
  3. 3Append that token and run a forward pass that reuses the cached keys and values instead of recomputing the prefix.
  4. 4Stop when the model emits an end-of-sequence token, matches a configured stop string, or reaches the output token cap.
  5. 5Detokenise the accumulated ids once at the end, since tokens do not align with word or character boundaries.
  6. 6Record prompt tokens, completion tokens and finish reason per request, because those three fields explain most cost and truncation incidents.

Step 1 of 6

Tokenise the full prompt and run a prefill pass that computes keys and values for every prompt position in one parallel operation.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Tokenise the full prompt and run a prefill pass that computes keys and values for every p… 2. Read the distribution at the final position, apply the decoding rule, and select one toke… 3. Append that token and run a forward pass that reuses the cached keys and values instead o… 4. Stop when the model emits an end-of-sequence token, matches a configured stop string, or … 5. Detokenise the accumulated ids once at the end, since tokens do not align with word or ch… 6. Record prompt tokens, completion tokens and finish reason per request, because those thre…

  1. Tokenise the full prompt and run a prefill pass that computes keys and values for every prompt position in one parallel operation.
  2. Read the distribution at the final position, apply the decoding rule, and select one token id.
  3. Append that token and run a forward pass that reuses the cached keys and values instead of recomputing the prefix.
  4. Stop when the model emits an end-of-sequence token, matches a configured stop string, or reaches the output token cap.
  5. Detokenise the accumulated ids once at the end, since tokens do not align with word or character boundaries.
  6. Record prompt tokens, completion tokens and finish reason per request, because those three fields explain most cost and truncation incidents.

Trade-offs

ChoiceGood forCostFails when
Single-pass answerLow latency, simple clientNo chance to check intermediate workThe task needs verification between steps
Streamed token loopEarly feedback, responsive feelPartial output must be handled and sometimes discardedDownstream parsing needs the complete document
Long outputsDetailed answers, worked reasoningDecode time and cost grow with every tokenThe budget is interactive latency
Reuse the KV cacheCheap continuation of a prefixMemory held per active sequenceConcurrency is high and sequences are long
Trade-off mapInteractive

Good for

Low latency, simple client

Cost

No chance to check intermediate work

Fails when

The task needs verification between steps

  1. The task needs verification between steps
  2. Downstream parsing needs the complete document
  3. The budget is interactive latency
  4. Concurrency is high and sequences are long

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Single-pass answer is good for Low latency, simple client, costs No chance to check intermediate work, and fails when The task needs verification between steps. Streamed token loop is good for Early feedback, responsive feel, costs Partial output must be handled and sometimes discarded, and fails when Downstream parsing needs the complete document. Long outputs is good for Detailed answers, worked reasoning, costs Decode time and cost grow with every token, and fails when The budget is interactive latency. Reuse the KV cache is good for Cheap continuation of a prefix, costs Memory held per active sequence, and fails when Concurrency is high and sequences are long

Failure modes

  • Prefix drift. A wrong early token makes the rest plausible but wrong, and the model cannot go back. Debug: compare the answer with a run that constrains the first sentence, and check whether the error is already present in the first few tokens.
  • Truncated finish. The response stops at the token cap mid-sentence and the client treats it as complete. Debug: log the finish reason and assert it is a natural stop before parsing.
  • Recomputed prefix. The server reprocesses the whole prompt on every call because caching is off, so latency scales with prompt length. Debug: compare time to first token for a short and a long prompt with the same output length.

In production

Watch time to first token and tokens per second separately, cap output tokens per route, and alert on the share of responses that end by length rather than by a stop condition.

Common misconceptions

People think: The model writes the whole answer and then returns it.

Actually: It commits to each token before seeing the next, so the answer is built incrementally and cannot be revised.

People think: A bigger model removes the need for output limits.

Actually: Every extra token is another forward pass; limits bound latency and cost regardless of model size.

Interview angle

Explain the loop, then connect it to observable behaviour: why latency grows with output length, why an early error is unrecoverable, and why prefill and decode respond to different fixes. A weak answer repeats next-token prediction as a slogan and cannot say what happens between two adjacent tokens.

Sources