Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 15 of 22

Voice agents

0 of 22 complete

Why it matters

A text agent can take seconds and show a spinner. A voice agent that pauses too long has already failed, because silence reads as a dropped call. That constraint compresses every stage: recognition must emit partials, the model must start responding before it has finished thinking, and tools must return fast or be announced. The loop has to sound like a conversation in its worst case.

Mental model

A voice agent is a phone call with a competent colleague who can look things up: the content matters, and so does the pause before the answer. The metaphor breaks because the colleague knows when to say one moment, while an agent must be taught to fill silence deliberately.

How it works

Two seconds of dead airInteractive

The interactive waveform loads with JavaScript. The text version below describes the same frame.

Figure. A synthetic 4.60-second call: a 1.2 s user turn, a 2.0 s blocking lookup, a spoken filler, and replies at 2.10 s without the tool and 4.10 s with it. A barge-in at 3.05 s stops playback at 3.15 s. All timings are authored; no telephony audio is used and nothing is played.

What to notice. Time to first audio is the number the caller feels: 0.60 s when the answer is local and 2.60 s when a lookup blocks, of which 2.00 s is the tool. A filler covers the wait only if it can be cancelled; a barge-in must cancel generation as well as audio, and the 100 ms between detection and silence is audible even when it is fast.

Text version

A synthetic 4.60-second call. Lanes show the user turn and a later barge-in, a hatched 2.0 second lookup, a 1.45 second spoken filler that stops at 3.15 seconds, a reply at 2.10 seconds without the tool, a reply at 4.10 seconds after it, and a playback stop from 3.05 to 3.15 seconds. Steps cover the end-of-turn clock, the dead air, the filler, the cancellation latency, and the 2.00 second difference between the two timelines.

  1. Draw the loop with a latency budget per hop and a target for time to first audio.
  2. Decide which stages stream, since a single non-streaming stage sets the floor for the whole conversation.
  3. Design the filler and escalation policy for slow tools, so a lookup becomes a spoken acknowledgement rather than silence.
  4. Handle barge-in and cancellation, including cancelling in-flight tool calls when the user interrupts.
  5. Make error recovery spoken and specific, with a fallback path to a human or a callback.
  6. Rehearse with real telephony audio, accents and background noise, not with clean studio recordings.

Trade-offs

ChoiceGood forCostFails when
Cascade of specialised modelsInspectable and debuggableLatency adds up across stagesEvery millisecond counts
End-to-end speech modelNatural timingOpaque and harder to constrainYou need auditable content
Streaming everythingFastest first responsePartial-context errorsAccuracy dominates
Turn-based with a fillerMore accurate answersFeels slowerThe task is quick lookup
Trade-off mapInteractive

Good for

Inspectable and debuggable

Cost

Latency adds up across stages

Fails when

Every millisecond counts

  1. Every millisecond counts
  2. You need auditable content
  3. Accuracy dominates
  4. The task is quick lookup

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Cascade of specialised models is good for Inspectable and debuggable, costs Latency adds up across stages, and fails when Every millisecond counts. End-to-end speech model is good for Natural timing, costs Opaque and harder to constrain, and fails when You need auditable content. Streaming everything is good for Fastest first response, costs Partial-context errors, and fails when Accuracy dominates. Turn-based with a filler is good for More accurate answers, costs Feels slower, and fails when The task is quick lookup

Failure modes

  • Dead air. A tool call blocks and the caller hears silence, then hangs up. Debug: measure the gap between end of user speech and first audio for sessions that include tool calls.
  • Interruption ignored. The agent keeps talking after the user starts and both are unintelligible. Debug: log barge-in events against playback state and check whether cancellation stopped the audio.
  • Recognition drift on telephony. Accuracy collapses on narrowband audio after tuning on wideband. Debug: evaluate on a sample recorded through the real telephony path.

In production

Monitor time to first audio, dead-air duration and call completion rate per intent. Rehearse the escalation path, since an agent that cannot hand off fails the caller twice.

Common misconceptions

People think: Voice agents are chatbots with speech bolted on.

Actually: The latency and turn-taking constraints change the architecture, not just the interface.

People think: A better model fixes a slow agent.

Actually: Perceived speed comes from streaming and turn policy as much as from model quality.

Interview angle

Design the loop with a budget per hop, then say which stage streams and what happens when a tool is slow. A strong answer treats interruption and cancellation as first-class. Weak answers describe a speech interface over a chat model and never mention dead air.

Sources