Why it matters
A long answer can take many seconds to generate, and during that time a non-streaming client shows nothing. Users read that silence as failure. Streaming exposes the generation loop directly, so the first token arrives after prefill and each later token shortly after it. The engineering cost is that partial states must be rendered, cancellations must be handled, and the final object is no longer available before display.
Mental model
Streaming is dictation versus handing over a finished letter: the listener starts understanding immediately but hears every hesitation. The analogy breaks because the model never corrects, so a wrong opening is displayed and then built upon rather than revised. A finished letter can be checked before it is sent; a stream cannot.
How it works
Event · step 1 of 5
0.3 s · prefill holds the turn
At 0.3 s
The client has sent the prompt and the provider is still in prefill. Nothing is on screen yet. For a non-streaming client this whole interval would look the same as a failure.
On screen at this event
- wait
- request
- prefill
- blank
Next at 0.62 s: 0.62 s · the first token is painted
- TTFT
- —
- Screen
- blank
- Upstream
- open
- After leave
- 0
Figure. A synthetic 2.4 s generation: prefill to a first token at 0.62 s, decode to 2.4 s, and a client that disconnects at 1.5 s while the upstream call keeps running. Times are illustrative.
What to notice. Streaming changes when output appears, not how long generation takes. The first token at 0.62 s is 26% of the total, which is the number the reader feels. The failure is after the leave: the gateway stops writing downstream but never aborts upstream, so 0.9 s of generation is billed and displayed nowhere, and no terminal event distinguishes a finished answer from a dropped connection.
Text version
Five lanes over 2.6 seconds: client, gateway, provider, model and screen. The client waits 0 to 0.62, reads 0.62 to 1.5, then a dashed gone span 1.5 to 2.6. The gateway streams 0 to 1.5 and shows a dashed still open 1.5 to 2.4. The provider request runs 0.04 to 2.4. Prefill runs 0.06 to 0.62 and decode 0.62 to 2.4. The screen is blank 0 to 0.62, tokens 0.62 to 1.5, then red stale text 1.5 to 2.6. Markers read first token at 0.62, client leaves at 1.5 and upstream ends at 2.4; no terminal event reaches the client. Counters: TTFT 0.62 s from step 2; screen blank, tokens, then stale; upstream open until 2.4; 0.9 s billed after the leave.
- Request a stream from the provider and read server-sent events incrementally rather than buffering the whole body.
- Emit deltas to the client as they arrive, and keep an accumulating buffer for logging and final validation.
- Render a stable reserved marker for tool calls and structured sections, since those may only be complete at the end.
- Handle cancellation: close the upstream request when the client disconnects so generation and cost stop.
- Defer validation and parsing until the stream ends, and show a distinct state if validation fails after display.
- Measure time to first token separately from total duration, because they respond to different fixes.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Token streaming | Fast perceived response | Partial output handling and harder parsing | The client needs a validated object before acting |
| Buffered response | Simple validation and retries | Long silence, worse perceived latency | Answers are long and interactive |
| Stream then validate | Responsive and checked | User may see content later retracted | A retraction is unacceptable in the domain |
| Stream with incremental parsing | Early structure and progress | Parser must handle fragments and errors | The format is not prefix-parseable |
Good for
Fast perceived response
Cost
Partial output handling and harder parsing
Fails when
The client needs a validated object before acting
- The client needs a validated object before acting
- Answers are long and interactive
- A retraction is unacceptable in the domain
- The format is not prefix-parseable
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Token streaming is good for Fast perceived response, costs Partial output handling and harder parsing, and fails when The client needs a validated object before acting. Buffered response is good for Simple validation and retries, costs Long silence, worse perceived latency, and fails when Answers are long and interactive. Stream then validate is good for Responsive and checked, costs User may see content later retracted, and fails when A retraction is unacceptable in the domain. Stream with incremental parsing is good for Early structure and progress, costs Parser must handle fragments and errors, and fails when The format is not prefix-parseable
Failure modes
- Partial JSON. The client tries to parse an incomplete object and fails on every chunk. Debug: log the raw chunks and confirm the parser only runs at stream end or uses an incremental parser.
- Orphaned generation. The client disconnects but the upstream request keeps running and billing. Debug: assert that disconnect handlers abort the provider request and check active request counts.
- Error after display. A mid-stream failure leaves truncated text on screen with no signal. Debug: emit an explicit terminal event and test a forced provider error mid-stream.
In production
Track time to first token at p50 and p95, plus aborted-request rate. Cap stream duration so a runaway generation cannot hold a connection open indefinitely.
Common misconceptions
People think: Streaming makes responses faster.
Actually: It makes the first visible output arrive sooner; total generation time is unchanged or slightly higher.
People think: Streaming and structured outputs are incompatible.
Actually: They compose if you validate at the end or parse incrementally, but partial text cannot be treated as a finished object.
Interview angle
Separate perceived latency from throughput and explain what streaming does to the failure model. Mention cancellation, terminal events and why parsing moves to the end. A weak answer says streaming is faster and does not address partial output or abandoned upstream requests.
Sources
- WHATWG, Server-sent events — the event stream format most providers use.
- Anthropic, Streaming messages — event types and incremental content blocks.
- Pope et al., Efficiently Scaling Transformer Inference — why the first token is expensive and later tokens are cheap.