Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 17 of 28

Streaming

0 of 28 complete

Why it matters

A long answer can take many seconds to generate, and during that time a non-streaming client shows nothing. Users read that silence as failure. Streaming exposes the generation loop directly, so the first token arrives after prefill and each later token shortly after it. The engineering cost is that partial states must be rendered, cancellations must be handled, and the final object is no longer available before display.

Mental model

Streaming is dictation versus handing over a finished letter: the listener starts understanding immediately but hears every hesitation. The analogy breaks because the model never corrects, so a wrong opening is displayed and then built upon rather than revised. A finished letter can be checked before it is sent; a stream cannot.

How it works

The first token takes 0.62 s; the reader leaves at 1.5 sInteractive
workwaitriskoff the client pathevent
0.3 s

Event · step 1 of 5

0.3 s · prefill holds the turn

At 0.3 s

The client has sent the prompt and the provider is still in prefill. Nothing is on screen yet. For a non-streaming client this whole interval would look the same as a failure.

On screen at this event

  • wait
  • request
  • prefill
  • blank

Next at 0.62 s: 0.62 s · the first token is painted

TTFT
—
Screen
blank
Upstream
open
After leave
0
Step 1/5 · 0.3 s · prefill holds the turn

Figure. A synthetic 2.4 s generation: prefill to a first token at 0.62 s, decode to 2.4 s, and a client that disconnects at 1.5 s while the upstream call keeps running. Times are illustrative.

What to notice. Streaming changes when output appears, not how long generation takes. The first token at 0.62 s is 26% of the total, which is the number the reader feels. The failure is after the leave: the gateway stops writing downstream but never aborts upstream, so 0.9 s of generation is billed and displayed nowhere, and no terminal event distinguishes a finished answer from a dropped connection.

Text version

Five lanes over 2.6 seconds: client, gateway, provider, model and screen. The client waits 0 to 0.62, reads 0.62 to 1.5, then a dashed gone span 1.5 to 2.6. The gateway streams 0 to 1.5 and shows a dashed still open 1.5 to 2.4. The provider request runs 0.04 to 2.4. Prefill runs 0.06 to 0.62 and decode 0.62 to 2.4. The screen is blank 0 to 0.62, tokens 0.62 to 1.5, then red stale text 1.5 to 2.6. Markers read first token at 0.62, client leaves at 1.5 and upstream ends at 2.4; no terminal event reaches the client. Counters: TTFT 0.62 s from step 2; screen blank, tokens, then stale; upstream open until 2.4; 0.9 s billed after the leave.

  1. Request a stream from the provider and read server-sent events incrementally rather than buffering the whole body.
  2. Emit deltas to the client as they arrive, and keep an accumulating buffer for logging and final validation.
  3. Render a stable reserved marker for tool calls and structured sections, since those may only be complete at the end.
  4. Handle cancellation: close the upstream request when the client disconnects so generation and cost stop.
  5. Defer validation and parsing until the stream ends, and show a distinct state if validation fails after display.
  6. Measure time to first token separately from total duration, because they respond to different fixes.

Trade-offs

ChoiceGood forCostFails when
Token streamingFast perceived responsePartial output handling and harder parsingThe client needs a validated object before acting
Buffered responseSimple validation and retriesLong silence, worse perceived latencyAnswers are long and interactive
Stream then validateResponsive and checkedUser may see content later retractedA retraction is unacceptable in the domain
Stream with incremental parsingEarly structure and progressParser must handle fragments and errorsThe format is not prefix-parseable
Trade-off mapInteractive

Good for

Fast perceived response

Cost

Partial output handling and harder parsing

Fails when

The client needs a validated object before acting

  1. The client needs a validated object before acting
  2. Answers are long and interactive
  3. A retraction is unacceptable in the domain
  4. The format is not prefix-parseable

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Token streaming is good for Fast perceived response, costs Partial output handling and harder parsing, and fails when The client needs a validated object before acting. Buffered response is good for Simple validation and retries, costs Long silence, worse perceived latency, and fails when Answers are long and interactive. Stream then validate is good for Responsive and checked, costs User may see content later retracted, and fails when A retraction is unacceptable in the domain. Stream with incremental parsing is good for Early structure and progress, costs Parser must handle fragments and errors, and fails when The format is not prefix-parseable

Failure modes

  • Partial JSON. The client tries to parse an incomplete object and fails on every chunk. Debug: log the raw chunks and confirm the parser only runs at stream end or uses an incremental parser.
  • Orphaned generation. The client disconnects but the upstream request keeps running and billing. Debug: assert that disconnect handlers abort the provider request and check active request counts.
  • Error after display. A mid-stream failure leaves truncated text on screen with no signal. Debug: emit an explicit terminal event and test a forced provider error mid-stream.

In production

Track time to first token at p50 and p95, plus aborted-request rate. Cap stream duration so a runaway generation cannot hold a connection open indefinitely.

Common misconceptions

People think: Streaming makes responses faster.

Actually: It makes the first visible output arrive sooner; total generation time is unchanged or slightly higher.

People think: Streaming and structured outputs are incompatible.

Actually: They compose if you validate at the end or parse incrementally, but partial text cannot be treated as a finished object.

Interview angle

Separate perceived latency from throughput and explain what streaming does to the failure model. Mention cancellation, terminal events and why parsing moves to the end. A weak answer says streaming is faster and does not address partial output or abandoned upstream requests.

Sources