Why it matters
A long generation can take many seconds, and users judge responsiveness by when text appears. Streaming sends tokens as they are produced, so the interface becomes usable earlier. The cost is a more complex contract: partial output, mid-stream errors, reconnection and cancellation all become states the system must handle, and most of them are invisible in a demo. The pattern is therefore as much about failure handling as about speed.
Mental model
Dictation versus a written report: you hear the answer as it forms, and you can interrupt it mid-sentence. The metaphor breaks because dictation can be resumed cleanly, while a stream that errors halfway leaves the client holding text the server never finished.
How it works
Step 1 of 6
Generate one step
The model emits a token at a time. Nothing about the total work or cost changes with streaming; only the moment the reader sees something does.
Active layers
- Model generates
Hover a layer to read what it holds.
Next: Emit typed events
- First token
- the metric users feel
- Events
- token · tool · error · done
- Cancel
- propagated to work
Figure. A stream from generation to the reader. The client's text arrives by the dashed edge without waiting for the assembled copy, which the server writes on completion. Event payload schemas, reconnection and proxy configuration are omitted; a buffering proxy is described in the steps rather than drawn.
What to notice. The dashed edge is why a mid-stream error leaves unfinished text on screen: the reader holds text the server never assembled and may never save. The dash-dot edge is cancellation, and it exists only if the disconnect is propagated; otherwise generation runs to completion for nobody. Streaming changes when the first token arrives, not the total work.
Text version
A vertical stack for a streamed response. The model generates one token per step, the application server turns the stream into typed events, a proxy that must not buffer carries chunks, and the client reader renders each chunk. The reader's text reaches the output along a dashed lane without waiting for the assembled message, which the server writes on completion and then commits to the output. A dash-dot lane runs from the client reader back to the model, labelled cancel on disconnect. Six steps cover generation, typed events, unbuffered transport, incremental rendering, persistence on completion, and cancellation.
- Choose the transport from the client: server-sent events for one-way token streams, WebSocket when the client must also send mid-stream.
- Define the event types before implementation: token delta, tool call, error, and completion with a finish reason.
- Disable proxy and compression buffering on the streaming route, and verify with a timed test that chunks arrive incrementally.
- Propagate client disconnects to the generation call so work stops when nobody is listening.
- Persist the assembled message at the end, and mark incomplete messages explicitly when the stream errors.
- Measure time to first token and inter-token latency separately from total duration.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Server-sent events | Simple, HTTP-native, auto-reconnect | One-way only | The client must send mid-stream |
| WebSocket | Bidirectional and low overhead | Connection management and scaling | A simple token stream is enough |
| Stream and persist at the end | Simple storage | Incomplete messages on error | Partial output must be resumable |
| Stream and persist per chunk | Recoverable | Write amplification | Storage is expensive |
Good for
Simple, HTTP-native, auto-reconnect
Cost
One-way only
Fails when
The client must send mid-stream
- The client must send mid-stream
- A simple token stream is enough
- Partial output must be resumable
- Storage is expensive
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Server-sent events is good for Simple, HTTP-native, auto-reconnect, costs One-way only, and fails when The client must send mid-stream. WebSocket is good for Bidirectional and low overhead, costs Connection management and scaling, and fails when A simple token stream is enough. Stream and persist at the end is good for Simple storage, costs Incomplete messages on error, and fails when Partial output must be resumable. Stream and persist per chunk is good for Recoverable, costs Write amplification, and fails when Storage is expensive
Failure modes
- Buffered stream. The client receives everything at once after a delay because an intermediary buffers. Debug: timestamp chunk arrivals at the client and check for a single burst.
- Half-written message. The stream errors and the client displays text that was never completed or saved. Debug: compare the persisted message with the rendered transcript and check the finish reason.
- Orphaned generation. The user closes the tab and the model keeps generating. Debug: count cancellations against disconnects and inspect provider usage for runs with no client.
In production
Monitor time to first token, completion rate and cancellation rate per route. Cap output length and set a server-side deadline so a stuck stream cannot run indefinitely.
Common misconceptions
People think: Streaming makes generation faster.
Actually: It makes the first token arrive sooner; total work and cost are unchanged.
People think: If the client disconnects, the server stops.
Actually: Only if cancellation is propagated; otherwise the generation runs to completion.
Interview angle
Discuss perceived latency, the event contract and mid-stream failure. Strong answers mention buffering intermediaries and cancellation propagation. Weak ones add streaming as a flag and never consider what the client holds when the stream breaks.
Sources
- WHATWG, Server-Sent Events — the event-stream format and reconnection semantics.
- Azure Architecture Center, Asynchronous Request-Reply — long-running work behind a request and how clients poll or stream.
- MDN, Streams API — incremental reading and backpressure in the browser.