Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 14 of 22

Streaming audio

0 of 22 complete

Why it matters

Batch pipelines can wait for the whole file. A live conversation cannot, so audio arrives in chunks and every component must produce output before the input is complete. That constraint changes the model interface, the buffering strategy and the error handling, because a dropped chunk is not a retryable request but a hole in the audio. The pipeline has to stay incremental end to end.

Mental model

Streaming audio is reading a sentence as it is written rather than waiting for the paragraph: you commit early and sometimes revise. The metaphor breaks because audio has real time attached, so a late correction arrives after the listener has already heard the mistake.

How it works

Chunks, partials and a hole in the streamInteractive

The interactive waveform loads with JavaScript. The text version below describes the same frame.

Figure. A synthetic 3.2-second stream in 200 ms chunks, with authored partial hypotheses at 1.2 s and 2.0 s, a final result at 3.2 s, and one lost chunk. Stable, uncertain and revised labels are teaching states, not calibrated posteriors; no audio is played.

What to notice. Latency and context trade directly: the chunk size sets the earliest possible result, while the stable prefix decides whether a partial can be shown without flicker. The final commits “fourteen” after the partial's “four”. A lost chunk is not a retryable request — the missing 200 ms is a hole, and the system can only drop or conceal it.

Text version

A synthetic 3.2-second stream. Sixteen 200 ms chunks share a timeline with a playhead; a second lane marks one lost chunk from 1.6 to 1.8 seconds. Steps show the chunk grid, the first partial at 1.2 seconds with two stable words and two uncertain ones, the second partial at 2.0 seconds with four stable words and a revised four, the final text with fourteen minutes, the lost chunk, and the slowest-stage delay. A scrubber moves across all lanes.

  1. Set a latency budget and derive the maximum chunk size each stage may buffer.
  2. Choose frame sizes aligned across capture, encoding and model input to avoid partial-frame handling.
  3. Design for partial hypotheses with a stable prefix, so the consumer can revise text without rewriting everything.
  4. Add a jitter buffer and sequence numbers, and decide explicitly whether to drop or conceal a lost packet.
  5. Measure time to first partial result and time to final result separately, since they drive different experience decisions.
  6. Test under simulated packet loss and variable network delay, not only on a clean local connection.

Trade-offs

ChoiceGood forCostFails when
Small chunksLow latencyMore partial errors and overheadAccuracy is critical
Large chunksBetter context and accuracyHigher latencyThe conversation feels laggy
Drop on lossKeeps up with real timeAudible gapsContent matters more than timing
Conceal on lossSmoother audioIntroduced artefactsSpeech recognition consumes the stream
Trade-off mapInteractive

Good for

Low latency

Cost

More partial errors and overhead

Fails when

Accuracy is critical

  1. Accuracy is critical
  2. The conversation feels laggy
  3. Content matters more than timing
  4. Speech recognition consumes the stream

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Small chunks is good for Low latency, costs More partial errors and overhead, and fails when Accuracy is critical. Large chunks is good for Better context and accuracy, costs Higher latency, and fails when The conversation feels laggy. Drop on loss is good for Keeps up with real time, costs Audible gaps, and fails when Content matters more than timing. Conceal on loss is good for Smoother audio, costs Introduced artefacts, and fails when Speech recognition consumes the stream

Failure modes

  • Buffer bloat. Latency grows during a session as queues fill. Debug: log queue depth per stage over a five-minute session and find the stage whose depth only rises.
  • Partial result thrash. The transcript rewrites earlier words and the interface flickers. Debug: diff consecutive partials and count how often the stable prefix shrinks.
  • Clock drift. Capture and playback clocks differ and the buffer slowly drains or fills. Debug: plot buffer occupancy against wall time for a long session.

In production

Alert on time to first partial result and on buffer occupancy trends. Rehearse loss and reconnection paths, since a streaming session that cannot recover is worse than one that ends cleanly.

Common misconceptions

People think: Streaming is batch processing with a smaller chunk.

Actually: Every stage must be incremental and revision-aware, which changes interfaces and error handling.

People think: Lower chunk size always improves experience.

Actually: It raises partial errors and overhead, and unstable text is worse than a short wait.

Interview angle

Walk the pipeline stage by stage and assign each a buffer and a latency budget. A strong answer distinguishes time to first partial from time to final and says how a lost packet is handled. Weak answers describe chunking as a transport detail and never mention partial-result revision.

Sources