Skip to content
AI Engineer Guide.
seniorReportedcurated

Budget the latency for a real-time voice agent: VAD, ASR, LLM, TTS, network. Where does the time go, and what do you optimise first?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Callers keep getting cut off mid-sentence. What is wrong and how do you fix it?
  • The endpointing threshold is too short for natural pauses, so the system treats a mid-sentence breath as end of turn.
  • Replace fixed silence with a semantic end-of-turn model, raise the silence window, and let barge-in interrupt playback rather than the microphone.
  • Measure false cut-off rate on recorded calls and track it beside time to first audio, because improving one by worsening the other is not a fix.
How does sentence chunking feed the LLM output into TTS sooner?
  • Stream the LLM output and flush at the first clause boundary, so TTS starts synthesising before the full answer exists.
  • Keep a small lookahead so prosody does not collapse, and accept revision risk when a later clause changes the sentence.
  • Cap the chunk size and measure time to first audio separately from total audio duration, since chunking moves the former and not the latter.

Red flags

  • Optimises total response time and ignores time-to-first-audio
  • Names no concrete stage durations or target budget

Concepts this tests

Tags

Source: github.com

More in Multimodal, vision & voice