Skip to content
AI Engineer Guide.
seniorReportedcurated

How do you evaluate ASR quality beyond word error rate, and TTS quality when there is no single correct output?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How would you catch a model putting words in the wrong speaker's mouth?
  • Diarisation is a separate model from ASR, so score speaker-attributed WER and per-speaker confusion rather than only the word stream.
  • Check turn boundaries against the audio and count swaps where one speaker's words are assigned to another.
  • Use per-segment confidence and flag low-confidence spans for review, because speaker errors are expensive in transcripts used as evidence.
Your WER improved but users complain more. What could explain that?
  • Entity and number errors may have risen while filler errors fell, so the average improved and the failures that matter got worse.
  • Latency or time to first audio may have regressed, which users feel even when the transcript is more accurate.
  • Normalisation, punctuation or speaker attribution may have changed, so run the sliced evaluation and compare task success, not only WER.

Red flags

  • Reports a single WER number as the whole evaluation
  • Uses MOS without saying who rated what, or how many raters

Concepts this tests

Tags

Source: github.com

More in Multimodal, vision & voice