Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 13 of 22

VAD

0 of 22 complete

Why it matters

Continuous audio is mostly silence or background. Running recognition on all of it wastes compute and produces hallucinated text from noise. A voice activity detector trims the stream, but it decides on a few milliseconds of signal, and both error directions are costly: a missed segment removes words the user said, and a spurious segment sends noise to a model that will try to transcribe it.

Mental model

Voice activity detection is a doorman deciding who enters: fast, cheap, and wrong at the margins. The metaphor breaks because a rejected guest cannot come back later, so a false negative permanently removes audio that no later stage can recover.

How it works

Hangover decides where the word endsInteractive

The interactive waveform loads with JavaScript. The text version below describes the same frame.

Figure. A synthetic 3.00-second phrase with a 140 ms gap between words and an 80 ms closure inside “report”, gated by four authored hangover policies. Each 60 ms span ends 60 ms after its sound, so the closure splits one word into two spans; at 250 ms the phrase is one span that ends at 1.99 s. Timings and spans are authored; no detector was run and no audio is played.

What to notice. Hangover is one knob with two costs: too short and the gate closes in every gap wider than the hangover, splitting words; too long and the endpoint arrives late by exactly the hangover. A gate that opens two 20 ms frames late deletes the first phoneme, and false negatives are the errors no later stage can undo. False positives only buy compute.

Text version

A synthetic 3.00-second phrase, send report, with a playhead. Lanes show the two words, a gate with 60 ms hangover whose three open spans each end 60 ms after their sound, a gate with 250 ms hangover that produces one span ending at 1.99 seconds, a gate that opens 40 ms late, and a gate with a lowered threshold that adds a false span from 0.10 to 0.26 seconds. Steps explain the bursts, the split word, the merged span, the clipped onset and the false span.

  1. Choose frame size from the latency budget, since the detector cannot decide faster than one frame.
  2. Add hangover so a brief dip in energy does not close the gate in the middle of a word.
  3. Calibrate on recordings from the actual device and environment, including the noisiest one you support.
  4. Set the threshold from the cost of each error, since missed speech is usually worse than extra compute.
  5. Run the detector before noise suppression as well as after, and compare, because suppression can remove quiet speech.
  6. Log the fraction of audio classified as speech to catch a threshold that has drifted.

Trade-offs

ChoiceGood forCostFails when
Energy thresholdTrivial and fastFails on noise and quiet speakersThe environment is uncontrolled
Neural VADRobust to noiseModel latency and tuningThe device is extremely constrained
Aggressive gatingLow computeDeletes quiet speechUsers speak softly or far from the microphone
Permissive gatingPreserves speechSends noise to the recogniserThe model hallucinates on noise
Trade-off mapInteractive

Good for

Trivial and fast

Cost

Fails on noise and quiet speakers

Fails when

The environment is uncontrolled

  1. The environment is uncontrolled
  2. The device is extremely constrained
  3. Users speak softly or far from the microphone
  4. The model hallucinates on noise

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Energy threshold is good for Trivial and fast, costs Fails on noise and quiet speakers, and fails when The environment is uncontrolled. Neural VAD is good for Robust to noise, costs Model latency and tuning, and fails when The device is extremely constrained. Aggressive gating is good for Low compute, costs Deletes quiet speech, and fails when Users speak softly or far from the microphone. Permissive gating is good for Preserves speech, costs Sends noise to the recogniser, and fails when The model hallucinates on noise

Failure modes

  • Clipped onsets. The first phoneme is removed because the gate opened late. Debug: compare transcripts of gated and ungated audio for the same clip and look for missing word starts.
  • Noise-triggered hallucination. The recogniser invents text from background noise. Debug: run the pipeline on known-silence recordings and check for non-empty transcripts.
  • Threshold drift after a device change. Speech detection rate falls and users report missing words. Debug: log speech fraction per device model and compare against the calibration baseline.

In production

Monitor the speech fraction and the empty-transcript rate together, since they move in opposite directions when the threshold is wrong. Re-calibrate per device class.

Common misconceptions

People think: VAD is a solved preprocessing step.

Actually: It is a threshold decision under noise, and its errors are irreversible for everything downstream.

People think: A neural VAD removes the need for tuning.

Actually: It still needs calibration on your devices and an explicit hangover policy.

Interview angle

Frame VAD as an asymmetric-cost decision: missed speech is unrecoverable, extra compute is not. A strong answer names frame size and hangover and says how they would calibrate on the real device. Weak answers treat it as a library call and never mention what happens to the audio it rejects.

Sources