Why it matters
A screen can show options, errors and history at once. Speech delivers one word at a time and then disappears, so the listener must hold everything in working memory. That changes what a good response is: shorter, ordered, and explicit about what happens next. Most voice failures are not model failures but design failures, where a correct answer arrives in a form nobody can follow.
Mental model
Voice UX is writing for radio rather than for a web page: the listener cannot scroll back or re-read a line. The metaphor breaks because radio is one-way, while a voice agent must also detect confusion and repair it in the next turn.
How it works
The interactive waveform loads with JavaScript. The text version below describes the same frame.
Figure. A synthetic 5.50-second session: a 0.72 s request, 1.48 s of silence, a four-item answer of 2.65 s end to end (2.37 s voiced), and a repeat request at 4.90 s, 0.20 s before the last item finishes. An earcon at 1.00 s and a two-turn answer are drawn as alternatives. All timings are authored and no audio is played.
What to notice. Speech is linear, so a correct four-item answer outruns a listener's memory and the repeat request is the measurable failure. Silence is interpreted: without an earcon, processing and a dropped line look the same. The repair is timing and state signalling, not better wording.
Text version
A synthetic 5.50-second session. Lanes show the request, a hatched 1.48 second silence, a four-item answer from 2.45 to 5.10 seconds, a repeat request from 4.90 to 5.30 seconds, a 120 ms earcon at 1.00 second, and an alternative answer in two chunks. Steps cover the linear answer, the ambiguous silence, the early repeat, the earcon, and the two-chunk answer.
- Write the response script for the top intents and read it aloud, since text that looks short often sounds long.
- Decide the confirmation policy: confirm irreversible actions, and skip confirmation for cheap reversible ones.
- Design the repair path first, with a way to repeat, slow down or escalate, because recognition errors are certain.
- Limit options per turn and order them by likelihood, since a spoken menu is remembered poorly.
- Use earcons or short phrases to mark listening, thinking and error states, so silence is never ambiguous.
- Test with people who have not seen the script, and record where they ask for repetition.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Verbose, explicit responses | Clear for complex tasks | Slow and tiring | The task is frequent and simple |
| Terse responses | Fast and natural | Ambiguous when something fails | Errors need explanation |
| Confirm every action | Fewer mistakes | Annoying for routine tasks | The action is reversible |
| No confirmation | Fluid | Irreversible mistakes | The action spends money or sends data |
Good for
Clear for complex tasks
Cost
Slow and tiring
Fails when
The task is frequent and simple
- The task is frequent and simple
- Errors need explanation
- The action is reversible
- The action spends money or sends data
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Verbose, explicit responses is good for Clear for complex tasks, costs Slow and tiring, and fails when The task is frequent and simple. Terse responses is good for Fast and natural, costs Ambiguous when something fails, and fails when Errors need explanation. Confirm every action is good for Fewer mistakes, costs Annoying for routine tasks, and fails when The action is reversible. No confirmation is good for Fluid, costs Irreversible mistakes, and fails when The action spends money or sends data
Failure modes
- Cognitive overload. Users ask for repetition after a long list. Debug: count repeat requests per response-length bucket and find where the rate rises.
- Ambiguous silence. Users say hello repeatedly because nothing signals processing. Debug: measure the gap before users re-prompt and check whether an earcon or phrase played.
- No repair path. A misheard name cannot be corrected and the session fails. Debug: review transcripts of failed sessions for repeated user corrections with no resolution.
In production
Track repeat-request rate, barge-in rate and task completion per intent. Review transcripts weekly, since voice failures are visible in the conversation before they appear in metrics.
Common misconceptions
People think: Good voice UX is a good language model.
Actually: Model quality does not fix responses that are too long, unordered or unrecoverable.
People think: Users prefer human-like small talk.
Actually: They prefer fast task completion; filler that delays the answer is a cost.
Interview angle
Design for a listener who cannot skim: short ordered responses, explicit state, and a repair path. A strong answer measures repeat requests and failed corrections. Weak answers focus on voice selection and persona while ignoring length, confirmation and recovery.
Sources
- W3C, Speech Synthesis Markup Language 1.1 — markup for pauses, emphasis and pronunciation in spoken output.
- Sacks, Schegloff & Jefferson, A Simplest Systematics for the Organization of Turn-Taking for Conversation — the conversational norms a spoken interface departs from.
- Ekstedt & Skantze, Voice Activity Projection — how systems can detect confusion and turn boundaries.