Skip to content
AI Engineer Guide.

pattern · architecture · intermediate · concept 6 of 7

Streaming pipeline

0 of 7 complete

Why it matters

A long generation can take many seconds, and users judge responsiveness by when text appears. Streaming sends tokens as they are produced, so the interface becomes usable earlier. The cost is a more complex contract: partial output, mid-stream errors, reconnection and cancellation all become states the system must handle, and most of them are invisible in a demo. The pattern is therefore as much about failure handling as about speed.

Mental model

Dictation versus a written report: you hear the answer as it forms, and you can interrupt it mid-sentence. The metaphor breaks because dictation can be resumed cleanly, while a stream that errors halfway leaves the client holding text the server never finished.

How it works

Three hops between the model and the eyeInteractive
signalbypassloop

Step 1 of 6

Generate one step

The model emits a token at a time. Nothing about the total work or cost changes with streaming; only the moment the reader sees something does.

Active layers

  • Model generates

Hover a layer to read what it holds.

Next: Emit typed events

First token
the metric users feel
Events
token · tool · error · done
Cancel
propagated to work
Step 1/6 · Generate one step

Figure. A stream from generation to the reader. The client's text arrives by the dashed edge without waiting for the assembled copy, which the server writes on completion. Event payload schemas, reconnection and proxy configuration are omitted; a buffering proxy is described in the steps rather than drawn.

What to notice. The dashed edge is why a mid-stream error leaves unfinished text on screen: the reader holds text the server never assembled and may never save. The dash-dot edge is cancellation, and it exists only if the disconnect is propagated; otherwise generation runs to completion for nobody. Streaming changes when the first token arrives, not the total work.

Text version

A vertical stack for a streamed response. The model generates one token per step, the application server turns the stream into typed events, a proxy that must not buffer carries chunks, and the client reader renders each chunk. The reader's text reaches the output along a dashed lane without waiting for the assembled message, which the server writes on completion and then commits to the output. A dash-dot lane runs from the client reader back to the model, labelled cancel on disconnect. Six steps cover generation, typed events, unbuffered transport, incremental rendering, persistence on completion, and cancellation.

  1. Choose the transport from the client: server-sent events for one-way token streams, WebSocket when the client must also send mid-stream.
  2. Define the event types before implementation: token delta, tool call, error, and completion with a finish reason.
  3. Disable proxy and compression buffering on the streaming route, and verify with a timed test that chunks arrive incrementally.
  4. Propagate client disconnects to the generation call so work stops when nobody is listening.
  5. Persist the assembled message at the end, and mark incomplete messages explicitly when the stream errors.
  6. Measure time to first token and inter-token latency separately from total duration.

Trade-offs

ChoiceGood forCostFails when
Server-sent eventsSimple, HTTP-native, auto-reconnectOne-way onlyThe client must send mid-stream
WebSocketBidirectional and low overheadConnection management and scalingA simple token stream is enough
Stream and persist at the endSimple storageIncomplete messages on errorPartial output must be resumable
Stream and persist per chunkRecoverableWrite amplificationStorage is expensive
Trade-off mapInteractive

Good for

Simple, HTTP-native, auto-reconnect

Cost

One-way only

Fails when

The client must send mid-stream

  1. The client must send mid-stream
  2. A simple token stream is enough
  3. Partial output must be resumable
  4. Storage is expensive

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Server-sent events is good for Simple, HTTP-native, auto-reconnect, costs One-way only, and fails when The client must send mid-stream. WebSocket is good for Bidirectional and low overhead, costs Connection management and scaling, and fails when A simple token stream is enough. Stream and persist at the end is good for Simple storage, costs Incomplete messages on error, and fails when Partial output must be resumable. Stream and persist per chunk is good for Recoverable, costs Write amplification, and fails when Storage is expensive

Failure modes

  • Buffered stream. The client receives everything at once after a delay because an intermediary buffers. Debug: timestamp chunk arrivals at the client and check for a single burst.
  • Half-written message. The stream errors and the client displays text that was never completed or saved. Debug: compare the persisted message with the rendered transcript and check the finish reason.
  • Orphaned generation. The user closes the tab and the model keeps generating. Debug: count cancellations against disconnects and inspect provider usage for runs with no client.

In production

Monitor time to first token, completion rate and cancellation rate per route. Cap output length and set a server-side deadline so a stuck stream cannot run indefinitely.

Common misconceptions

People think: Streaming makes generation faster.

Actually: It makes the first token arrive sooner; total work and cost are unchanged.

People think: If the client disconnects, the server stops.

Actually: Only if cancellation is propagated; otherwise the generation runs to completion.

Interview angle

Discuss perceived latency, the event contract and mid-stream failure. Strong answers mention buffering intermediaries and cancellation propagation. Weak ones add streaming as a flag and never consider what the client holds when the stream breaks.

Sources