seniorReportedcurated
Argue both sides: cascaded ASR plus LLM plus TTS versus a native speech-to-speech model for a production voice agent.
30-second answer
- A cascade gives a transcript you can log, redact and score with WER, lets you swap ASR, LLM and TTS vendors independently, and reuses the tools and guardrails already built for text.
- A native speech-to-speech model skips the text bottleneck, so it keeps tone, emphasis and timing and handles barge-in more naturally, at the cost of the transcript you would inspect.
- The costs invert: cascaded adds a hop per stage and loses paralinguistics, while native is opaque and locks you to one vendor's model and voice.
- For regulated or heavily instrumented products I start cascaded; for consumer conversational UX I start native and add an independent recogniser on the output for content auditing.
- Neither architecture removes the turn-taking problem, because deciding when the user has finished is a system policy that sits outside both.
Likely follow-ups
Open a question to rehearse the answer.
How would you evaluate the native model when there is no transcript to score?
- Run an independent recogniser over the output audio and score content accuracy against the expected answer, while treating that transcript as an eval artefact rather than product state.
- Use human ratings for naturalness, prosody and interruption handling, plus task success on scripted scenarios with a verified end state.
- Score scripted scenarios against a fixed expected answer, because a model that sounds fluent can still drop a price or a date in the audio.
Where exactly does the latency go in the cascaded version?
- Endpointing waits for silence or a semantic end-of-turn decision before ASR even finalises, and that wait is usually the largest single term.
- Streaming ASR adds partial-result latency, the LLM adds time to first token, and TTS adds time to first audio chunk, each with its own tail.
- Network round trips and audio buffering sit on top, so measure time to first audio per stage rather than the sum of averages.
Red flags
- Picks the native model because it sounds more advanced
- Claims cascaded is inherently low latency when each stage adds a hop
Concepts this tests
Tags
Source: github.com
More in Multimodal, vision & voice
- Budget the latency for a real-time voice agent: VAD, ASR, LLM, TTS, network. Where does the time go, and what do you optimise first?
- A video summariser samples one frame every three seconds. A customer summary says an export may not have finished, but the recording shows an Export ready toast. What went wrong, and how do you fix it?
- When would you use a multimodal model instead of OCR plus an extraction pipeline, and what are its failure modes?