Skip to content
AI Engineer Guide.

concept · concepts · advanced · concept 22 of 22

Voice UX

0 of 22 complete

Why it matters

A screen can show options, errors and history at once. Speech delivers one word at a time and then disappears, so the listener must hold everything in working memory. That changes what a good response is: shorter, ordered, and explicit about what happens next. Most voice failures are not model failures but design failures, where a correct answer arrives in a form nobody can follow.

Mental model

Voice UX is writing for radio rather than for a web page: the listener cannot scroll back or re-read a line. The metaphor breaks because radio is one-way, while a voice agent must also detect confusion and repair it in the next turn.

How it works

The repeat arrives before the answer endsInteractive

The interactive waveform loads with JavaScript. The text version below describes the same frame.

Figure. A synthetic 5.50-second session: a 0.72 s request, 1.48 s of silence, a four-item answer of 2.65 s end to end (2.37 s voiced), and a repeat request at 4.90 s, 0.20 s before the last item finishes. An earcon at 1.00 s and a two-turn answer are drawn as alternatives. All timings are authored and no audio is played.

What to notice. Speech is linear, so a correct four-item answer outruns a listener's memory and the repeat request is the measurable failure. Silence is interpreted: without an earcon, processing and a dropped line look the same. The repair is timing and state signalling, not better wording.

Text version

A synthetic 5.50-second session. Lanes show the request, a hatched 1.48 second silence, a four-item answer from 2.45 to 5.10 seconds, a repeat request from 4.90 to 5.30 seconds, a 120 ms earcon at 1.00 second, and an alternative answer in two chunks. Steps cover the linear answer, the ambiguous silence, the early repeat, the earcon, and the two-chunk answer.

  1. Write the response script for the top intents and read it aloud, since text that looks short often sounds long.
  2. Decide the confirmation policy: confirm irreversible actions, and skip confirmation for cheap reversible ones.
  3. Design the repair path first, with a way to repeat, slow down or escalate, because recognition errors are certain.
  4. Limit options per turn and order them by likelihood, since a spoken menu is remembered poorly.
  5. Use earcons or short phrases to mark listening, thinking and error states, so silence is never ambiguous.
  6. Test with people who have not seen the script, and record where they ask for repetition.

Trade-offs

ChoiceGood forCostFails when
Verbose, explicit responsesClear for complex tasksSlow and tiringThe task is frequent and simple
Terse responsesFast and naturalAmbiguous when something failsErrors need explanation
Confirm every actionFewer mistakesAnnoying for routine tasksThe action is reversible
No confirmationFluidIrreversible mistakesThe action spends money or sends data
Trade-off mapInteractive

Good for

Clear for complex tasks

Cost

Slow and tiring

Fails when

The task is frequent and simple

  1. The task is frequent and simple
  2. Errors need explanation
  3. The action is reversible
  4. The action spends money or sends data

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Verbose, explicit responses is good for Clear for complex tasks, costs Slow and tiring, and fails when The task is frequent and simple. Terse responses is good for Fast and natural, costs Ambiguous when something fails, and fails when Errors need explanation. Confirm every action is good for Fewer mistakes, costs Annoying for routine tasks, and fails when The action is reversible. No confirmation is good for Fluid, costs Irreversible mistakes, and fails when The action spends money or sends data

Failure modes

  • Cognitive overload. Users ask for repetition after a long list. Debug: count repeat requests per response-length bucket and find where the rate rises.
  • Ambiguous silence. Users say hello repeatedly because nothing signals processing. Debug: measure the gap before users re-prompt and check whether an earcon or phrase played.
  • No repair path. A misheard name cannot be corrected and the session fails. Debug: review transcripts of failed sessions for repeated user corrections with no resolution.

In production

Track repeat-request rate, barge-in rate and task completion per intent. Review transcripts weekly, since voice failures are visible in the conversation before they appear in metrics.

Common misconceptions

People think: Good voice UX is a good language model.

Actually: Model quality does not fix responses that are too long, unordered or unrecoverable.

People think: Users prefer human-like small talk.

Actually: They prefer fast task completion; filler that delays the answer is a cost.

Interview angle

Design for a listener who cannot skim: short ordered responses, explicit state, and a repair path. A strong answer measures repeat requests and failed corrections. Weak answers focus on voice selection and persona while ignoring length, confirmation and recovery.

Sources