Why it matters
A text agent can take seconds and show a spinner. A voice agent that pauses too long has already failed, because silence reads as a dropped call. That constraint compresses every stage: recognition must emit partials, the model must start responding before it has finished thinking, and tools must return fast or be announced. The loop has to sound like a conversation in its worst case.
Mental model
A voice agent is a phone call with a competent colleague who can look things up: the content matters, and so does the pause before the answer. The metaphor breaks because the colleague knows when to say one moment, while an agent must be taught to fill silence deliberately.
How it works
The interactive waveform loads with JavaScript. The text version below describes the same frame.
Figure. A synthetic 4.60-second call: a 1.2 s user turn, a 2.0 s blocking lookup, a spoken filler, and replies at 2.10 s without the tool and 4.10 s with it. A barge-in at 3.05 s stops playback at 3.15 s. All timings are authored; no telephony audio is used and nothing is played.
What to notice. Time to first audio is the number the caller feels: 0.60 s when the answer is local and 2.60 s when a lookup blocks, of which 2.00 s is the tool. A filler covers the wait only if it can be cancelled; a barge-in must cancel generation as well as audio, and the 100 ms between detection and silence is audible even when it is fast.
Text version
A synthetic 4.60-second call. Lanes show the user turn and a later barge-in, a hatched 2.0 second lookup, a 1.45 second spoken filler that stops at 3.15 seconds, a reply at 2.10 seconds without the tool, a reply at 4.10 seconds after it, and a playback stop from 3.05 to 3.15 seconds. Steps cover the end-of-turn clock, the dead air, the filler, the cancellation latency, and the 2.00 second difference between the two timelines.
- Draw the loop with a latency budget per hop and a target for time to first audio.
- Decide which stages stream, since a single non-streaming stage sets the floor for the whole conversation.
- Design the filler and escalation policy for slow tools, so a lookup becomes a spoken acknowledgement rather than silence.
- Handle barge-in and cancellation, including cancelling in-flight tool calls when the user interrupts.
- Make error recovery spoken and specific, with a fallback path to a human or a callback.
- Rehearse with real telephony audio, accents and background noise, not with clean studio recordings.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Cascade of specialised models | Inspectable and debuggable | Latency adds up across stages | Every millisecond counts |
| End-to-end speech model | Natural timing | Opaque and harder to constrain | You need auditable content |
| Streaming everything | Fastest first response | Partial-context errors | Accuracy dominates |
| Turn-based with a filler | More accurate answers | Feels slower | The task is quick lookup |
Good for
Inspectable and debuggable
Cost
Latency adds up across stages
Fails when
Every millisecond counts
- Every millisecond counts
- You need auditable content
- Accuracy dominates
- The task is quick lookup
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Cascade of specialised models is good for Inspectable and debuggable, costs Latency adds up across stages, and fails when Every millisecond counts. End-to-end speech model is good for Natural timing, costs Opaque and harder to constrain, and fails when You need auditable content. Streaming everything is good for Fastest first response, costs Partial-context errors, and fails when Accuracy dominates. Turn-based with a filler is good for More accurate answers, costs Feels slower, and fails when The task is quick lookup
Failure modes
- Dead air. A tool call blocks and the caller hears silence, then hangs up. Debug: measure the gap between end of user speech and first audio for sessions that include tool calls.
- Interruption ignored. The agent keeps talking after the user starts and both are unintelligible. Debug: log barge-in events against playback state and check whether cancellation stopped the audio.
- Recognition drift on telephony. Accuracy collapses on narrowband audio after tuning on wideband. Debug: evaluate on a sample recorded through the real telephony path.
In production
Monitor time to first audio, dead-air duration and call completion rate per intent. Rehearse the escalation path, since an agent that cannot hand off fails the caller twice.
Common misconceptions
People think: Voice agents are chatbots with speech bolted on.
Actually: The latency and turn-taking constraints change the architecture, not just the interface.
People think: A better model fixes a slow agent.
Actually: Perceived speed comes from streaming and turn policy as much as from model quality.
Interview angle
Design the loop with a budget per hop, then say which stage streams and what happens when a tool is slow. A strong answer treats interruption and cancellation as first-class. Weak answers describe a speech interface over a chat model and never mention dead air.
Sources
- OpenAI, Realtime API guide — streaming speech sessions with interruption support.
- Pipecat — an open framework for real-time voice pipelines.
- Defossez et al., Moshi — full-duplex dialogue without a separate turn boundary.