Why it matters
A completion that takes twelve seconds is unusable as a single response and tolerable as a stream, because the user starts reading after a few hundred milliseconds. That difference is not cosmetic: for chat, search summaries and code assistants it decides whether the interface feels responsive, and it changes what the user does while waiting — read, edit, cancel.
Streaming also changes the shape of the system. The server cannot set a Content-Length, cannot retry after partial output, and must handle clients that disconnect mid-generation. Those constraints are the source of most streaming bugs, and they are not visible in a non-streaming prototype.
Mental model
A radio broadcast on an open phone line. The server keeps talking; the client listens and processes words as they arrive rather than waiting for the whole speech. Nothing is requested twice, and there is no natural end except the speaker stopping.
The analogy breaks at reconnection. A radio listener who tunes in late misses the start; an SSE client that reconnects may be replayed from an event ID if the server keeps history, or may silently lose the middle of the sentence if it does not. Resumability is a design decision you must make, not a property you inherit.
How it works
Event · step 1 of 6
0.22 s · five frames written and flushed
At 0.22 s
The handler writes five token frames and flushes after each one. Locally the client would have painted text by now. Nothing has reached the browser, because the edge proxy is buffering the response.
On screen at this event
- f1–f5
Next at 0.7 s: 0.7 s · the heartbeat is written
- Frames written
- 5
- Frames parsed
- 0
- Chunks
- 0
- First byte
- —
- Stall
- 0
Figure. A synthetic 2.4 s stream: the handler writes and flushes five frames by 0.22 s, an edge proxy buffers 4 KB until 1.6 s, and the client reads three chunks in which one boundary falls inside frame 6. Times and buffer size are illustrative.
What to notice. The model is not the delay: the frames exist 1.4 s before the first byte arrives. A heartbeat written at 0.7 s is queued behind the same buffer, so it cannot keep an idle connection alive. The parser must split on the blank line and hold the incomplete frame: chunk 2 ends inside frame 6, and treating a chunk as a frame would emit truncated JSON. Disable proxy buffering and compression for the stream, and measure first byte and first frame separately.
Text version
Five lanes over 2.4 seconds: generator, edge proxy, client bytes, parser and screen. The generator writes frames f1 to f5 from 0.03 to 0.22, a heartbeat at 0.68 to 0.7, frame 6 at 1.1 to 1.14 and done at 1.45 to 1.48. The proxy holds a 4 KB buffer from 0 to 1.6 and flushes at 1.6 to 1.62. Client bytes are three chunks on three rows: chunk 1 1.6 to 1.63, chunk 2 1.63 to 1.66 and chunk 3 1.66 to 1.7. The parser emits f1 to f5 1.6 to 1.63, frame 6 1.66 to 1.68 and done 1.68 to 1.7. The screen is blank until 1.6, bursts 1.6 to 1.7 and reads to 2.4. Markers read first byte at 1.6 and split in frame 6 at 1.66, when chunk 2 ends. Counters: frames written 5, 5, 6, 6, 6, 6; frames parsed 0, 0, 0, 5, 6, 6; chunks 0, 0, 0, 2, 2, 3, because chunk 2 completes at 1.66; first byte 1.60 s from step 3; stall 1.38 s.
- Open one response and keep it open. The client sends a normal request, usually with
Accept: text/event-stream. The server replies200withContent-Type: text/event-streamand does not close the body. - Frame each message as text. A frame is a set of
field: valuelines terminated by a blank line.data:carries payload; multipledata:lines in one frame are joined with newlines.event:names the type,id:sets the resume cursor,retry:suggests a reconnect delay. A line starting with:is a comment, which is how heartbeats are sent. - Flush after every frame. Writing to a buffered response and never flushing is the classic bug: the client receives nothing until the handler returns. Disable response compression for the stream and turn off proxy buffering at the edge.
- Let the application define its own event vocabulary. Token deltas, tool-call starts, citations and errors all travel through the same channel. Include an explicit terminal event so the client can distinguish "finished" from "connection dropped".
- Parse by frame, not by chunk. Network chunks do not respect event boundaries. Buffer bytes, split on the blank-line delimiter, and only then decode the frame. A parser that assumes one chunk equals one event will corrupt output under load.
- Handle disconnect as cancellation. When the client goes away, the server should stop generation and release the upstream call. Otherwise you keep paying for tokens nobody reads.
- Decide the resume story. With
EventSource, the browser reconnects automatically and sendsLast-Event-ID. If you emit IDs and retain recent events, you can replay. If you do not, the client must restart the turn and the UI should say so.
A minimal stream
import json
import httpx
async def stream_reply(prompt: str):
async with httpx.stream(
"POST",
f"{BASE_URL}/generate",
json={"input": prompt, "stream": True},
headers={"accept": "text/event-stream"},
timeout=httpx.Timeout(60.0, connect=5.0),
) as resp:
resp.raise_for_status()
buffer = ""
async for chunk in resp.aiter_text():
buffer += chunk
while "\n\n" in buffer: # a frame ends at a blank line
frame, buffer = buffer.split("\n\n", 1)
for line in frame.splitlines():
if line.startswith("data:"):
payload = line[5:].strip()
if payload == "[DONE]":
return
yield json.loads(payload)Two details carry the correctness: the buffer survives chunk boundaries, and the terminal sentinel is explicit. Everything else is schema handling per event type.
Text version
The client opens a request, the server responds with event-stream headers, tokens arrive as data frames, and a sentinel closes the stream.
Trade-offs
| Transport | Good for | Cost | Fails when |
|---|---|---|---|
| SSE over HTTP | One-way token streams, simple clients, proxy-friendly | One connection per stream, text only | You need bidirectional messages mid-turn |
| WebSocket | Bidirectional, low-latency interaction | Connection state, heartbeats, no HTTP caching | Traffic is one-way and infrequent |
| Long polling | Old clients, simple infrastructure | Latency per poll, wasted requests | The response is long and continuous |
| Non-streaming JSON | Short answers, batch jobs | No perceived progress | A response takes more than a couple of seconds |
| Chunked JSON lines | Internal services, structured events | Non-standard framing, client must parse lines | Browsers or intermediaries are involved |
Good for
One-way token streams, simple clients, proxy-friendly
Cost
One connection per stream, text only
Fails when
You need bidirectional messages mid-turn
- You need bidirectional messages mid-turn
- Traffic is one-way and infrequent
- The response is long and continuous
- A response takes more than a couple of seconds
- Browsers or intermediaries are involved
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: SSE over HTTP is good for One-way token streams, simple clients, proxy-friendly, costs One connection per stream, text only, and fails when You need bidirectional messages mid-turn. WebSocket is good for Bidirectional, low-latency interaction, costs Connection state, heartbeats, no HTTP caching, and fails when Traffic is one-way and infrequent. Long polling is good for Old clients, simple infrastructure, costs Latency per poll, wasted requests, and fails when The response is long and continuous. Non-streaming JSON is good for Short answers, batch jobs, costs No perceived progress, and fails when A response takes more than a couple of seconds. Chunked JSON lines is good for Internal services, structured events, costs Non-standard framing, client must parse lines, and fails when Browsers or intermediaries are involved
Failure modes
Buffering in the middle. The stream works locally and arrives all at once in production because a proxy or compression layer buffers. Debug: curl the endpoint directly against the deployed URL and compare first-byte timing with a bypassed edge.
Missing flush. The framework buffers writes until the handler returns. Debug: log the time of the first yield and the time the client sees the first token; a large gap with a fast model points at the server.
Chunk-boundary parsing. Events are split across reads and the parser emits truncated JSON. Debug: replay a captured stream in one-byte pieces through the parser; any exception means the buffer logic is wrong.
Orphaned generations. The user closes the tab and the server keeps generating. Debug: count in-flight generations against open connections; they should track each other.
In production
Report time-to-first-token separately from total duration, plus tokens per second, stream completion rate and disconnect rate. A high disconnect rate can be a UI problem, a network problem or a model that takes too long before the first token — the timing distribution tells you which. Send a comment heartbeat inside the idle timeout of every proxy in the path. Cap the stream's maximum duration, and make sure the abort path cancels the upstream request rather than only closing the downstream connection. Then treat streaming errors as a distinct category: after the first token you cannot replay the request, so the only options are an explicit in-band error event and a client-side retry of the whole turn.
Common misconceptions
People think: Streaming reduces total latency.
Actually: It changes when the first useful output appears. Total generation time is unchanged, and sometimes slightly worse because each frame costs overhead.
People think: A dropped SSE connection is retried automatically by the browser, so nothing is lost.
Actually: EventSource reconnects, but only replay from an event ID recovers the missed messages. Without server-side history, the middle of the response is gone.
Interview angle
The question is usually framed as latency: "your chat app feels slow — what do you do?" The strong answer separates time-to-first-token from total duration, picks streaming as a perceived-latency fix, and then names the operational consequences: buffering, cancellation, partial-output errors, and one connection per active stream.
See latency and TTFT and production questions.
Sources
- HTML Living Standard: server-sent events — the normative event-stream grammar and
EventSourcebehaviour, including reconnection. - MDN: using server-sent events — field-by-field examples and client-side handling.
- OpenAI streaming responses guide — a hosted model API's streaming contract, including its terminal sentinel and event shape.
- RFC 9112: HTTP/1.1 — message framing and connection handling that explain why intermediaries break long responses.