seniorReportedcurated
Budget the latency for a real-time voice agent: VAD, ASR, LLM, TTS, network. Where does the time go, and what do you optimise first?
30-second answer
- Start from a target of roughly 800 ms to first audio and allocate backwards; the material fixes the target and the method but not the per-stage numbers, so measure each share at your own p95.
- Instrument endpointing first: a fixed silence threshold trades false cut-offs against dead air, while a semantic end-of-turn model spends compute to recover several hundred milliseconds of patience.
- After endpointing, the budget covers streaming ASR finalisation, LLM time to first token, TTS time to first chunk and one network round trip; measure each at its own p95, and treat the LLM leg as TTFT plus (N-1) times ITL.
- Optimise time to first audio before total duration, because a 400 ms first token and 25 ms per token makes a 300-token answer finish near 7.9 seconds while the user already judged responsiveness at the first syllable.
- Aggressive endpointing feels fast and interrupts people mid-thought, which is a worse failure than 200 ms of silence, so tune the threshold against recorded calls rather than the demo.
Likely follow-ups
Open a question to rehearse the answer.
Callers keep getting cut off mid-sentence. What is wrong and how do you fix it?
- The endpointing threshold is too short for natural pauses, so the system treats a mid-sentence breath as end of turn.
- Replace fixed silence with a semantic end-of-turn model, raise the silence window, and let barge-in interrupt playback rather than the microphone.
- Measure false cut-off rate on recorded calls and track it beside time to first audio, because improving one by worsening the other is not a fix.
How does sentence chunking feed the LLM output into TTS sooner?
- Stream the LLM output and flush at the first clause boundary, so TTS starts synthesising before the full answer exists.
- Keep a small lookahead so prosody does not collapse, and accept revision risk when a later clause changes the sentence.
- Cap the chunk size and measure time to first audio separately from total audio duration, since chunking moves the former and not the latter.
Red flags
- Optimises total response time and ignores time-to-first-audio
- Names no concrete stage durations or target budget
Concepts this tests
Tags
Source: github.com
More in Multimodal, vision & voice
- Argue both sides: cascaded ASR plus LLM plus TTS versus a native speech-to-speech model for a production voice agent.
- A video summariser samples one frame every three seconds. A customer summary says an export may not have finished, but the recording shows an Export ready toast. What went wrong, and how do you fix it?
- When would you use a multimodal model instead of OCR plus an extraction pipeline, and what are its failure modes?