seniorReportedcurated
How do you evaluate ASR quality beyond word error rate, and TTS quality when there is no single correct output?
30-second answer
- WER averages every word equally, so segment it: named entities, numbers, dates and product codes carry most of the product risk and one substitution there is worse than three filler errors.
- Score speaker attribution separately, add code-switching and accent slices, and measure end-to-end task success, because a booking either completes against the final state or it does not.
- For TTS, use paired preference tests on real sentences plus intelligibility checks on hard strings such as part numbers and dates, and track time to first audio and interruption handling.
- Human listening panels remain the gold standard and the most expensive instrument, so sample deliberately across channel, accent, noise level and domain rather than continuously.
- A single aggregate number hides the failure that matters, and a rising WER can coexist with better task success if the errors land on words the task does not use.
Likely follow-ups
Open a question to rehearse the answer.
How would you catch a model putting words in the wrong speaker's mouth?
- Diarisation is a separate model from ASR, so score speaker-attributed WER and per-speaker confusion rather than only the word stream.
- Check turn boundaries against the audio and count swaps where one speaker's words are assigned to another.
- Use per-segment confidence and flag low-confidence spans for review, because speaker errors are expensive in transcripts used as evidence.
Your WER improved but users complain more. What could explain that?
- Entity and number errors may have risen while filler errors fell, so the average improved and the failures that matter got worse.
- Latency or time to first audio may have regressed, which users feel even when the transcript is more accurate.
- Normalisation, punctuation or speaker attribution may have changed, so run the sliced evaluation and compare task success, not only WER.
Red flags
- Reports a single WER number as the whole evaluation
- Uses MOS without saying who rated what, or how many raters
Concepts this tests
Tags
Source: github.com
More in Multimodal, vision & voice
- Argue both sides: cascaded ASR plus LLM plus TTS versus a native speech-to-speech model for a production voice agent.
- Budget the latency for a real-time voice agent: VAD, ASR, LLM, TTS, network. Where does the time go, and what do you optimise first?
- A video summariser samples one frame every three seconds. A customer summary says an export may not have finished, but the recording shows an Export ready toast. What went wrong, and how do you fix it?