Skip to content
AI Engineer Guide.
seniorReportedcurated

Argue both sides: cascaded ASR plus LLM plus TTS versus a native speech-to-speech model for a production voice agent.

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How would you evaluate the native model when there is no transcript to score?
  • Run an independent recogniser over the output audio and score content accuracy against the expected answer, while treating that transcript as an eval artefact rather than product state.
  • Use human ratings for naturalness, prosody and interruption handling, plus task success on scripted scenarios with a verified end state.
  • Score scripted scenarios against a fixed expected answer, because a model that sounds fluent can still drop a price or a date in the audio.
Where exactly does the latency go in the cascaded version?
  • Endpointing waits for silence or a semantic end-of-turn decision before ASR even finalises, and that wait is usually the largest single term.
  • Streaming ASR adds partial-result latency, the LLM adds time to first token, and TTS adds time to first audio chunk, each with its own tail.
  • Network round trips and audio buffering sit on top, so measure time to first audio per stage rather than the sum of averages.

Red flags

  • Picks the native model because it sounds more advanced
  • Claims cascaded is inherently low latency when each stage adds a hop

Concepts this tests

Tags

Source: github.com

More in Multimodal, vision & voice