Why ASR sits in the critical path
Speech is the cheapest input channel for users and the most expensive one to process. A voice interface, a meeting transcript, a call-centre audit or a captioned video all depend on the same conversion, and everything downstream — retrieval, summarisation, agent actions — inherits its errors.
That inheritance is asymmetric. A dropped filler word is invisible to a summary. A misheard surname, a negated instruction, or a digit in an invoice number can invert the meaning or corrupt a record. ASR quality is therefore not one number; it is a distribution of error types, and the ones that matter depend on what consumes the text.
Mental model
A transcriptionist working from sound alone, with no ability to ask "did you say fifteen or fifty?". They write the most probable sentence consistent with what they heard, using the vocabulary and phrasing they have seen most often. That is why an unusual surname becomes a common word, and why a noisy passage becomes a fluent sentence that was never spoken.
The analogy breaks on self-knowledge. A human transcriptionist marks uncertainty or leaves a gap; a neural ASR system emits its best sequence with no reliable signal that it is guessing, unless you ask for per-token scores and calibrate them. It also has no access to the conversation's context beyond what you give it, so domain hints must be supplied deliberately.
How it works
Figure. A synthetic 4-second utterance: waveform, a schematic 24-band energy grid measured from the same samples, an authored VAD span (0.22–3.22 s, not detected here), and ten tokens with illustrative timestamps. No recording is used or played.
What to notice. Ten tokens is a deliberately tiny sample: one name substitution is already 10% word error rate, so the average hides nothing — the error is the whole story. In a real pipeline timestamps come from aligning the decoded text to the audio, not from the acoustic model alone.
Text version
Synthetic audio. A four-second waveform with an authored speech span from 0.22 to 3.22 seconds and a playhead. Below it, a schematic 24-band energy grid measured from the same waveform samples, brightest during speech and dimmer in the gaps between words. Below that, ten transcript tokens with illustrative timestamps: so, the, quarterly, report, goes, to, Priya, before, Friday, please. A slider scrubs the playhead and highlights the token under it. Switching noise on thickens the waveform floor, brightens every band, lowers several token confidences, and replaces Priya with the uncertain token Prya at 0.41 authored confidence.
- Capture and resample. Convert to mono at the model's expected sample rate (commonly 16 kHz), normalise level, and reject clipped or near-silent input early.
- Segment. Voice activity detection marks speech regions, trims long silences, and produces chunk boundaries. Bad segmentation is a common source of both latency and truncation.
- Extract features. Compute log-mel spectrogram frames over short windows with a small hop, which turns waveform into a time–frequency representation the encoder can consume.
- Encode acoustics. A transformer or convolution-augmented encoder turns frames into contextual hidden states. This is where noise robustness and speaker variation are absorbed.
- Decode. Three families dominate. CTC predicts a label per frame plus a blank, then collapses repeats — simple, non-autoregressive, strong with external language models. RNN-Transducer combines a prediction network with a joint network and streams naturally. Attention encoder–decoder models, including Whisper, generate tokens autoregressively from the encoded audio.
- Normalise. Detokenise, restore punctuation and casing, expand or contract numbers and dates to house style, and attach timestamps.
- Adapt. Apply custom vocabulary or biasing for names, products and jargon; optionally fine-tune on in-domain audio.
- Post-process. Diarisation, profanity masking, and redaction are separate models or rules layered on the transcript.
Whisper is the reference example for the batch case: a sequence-to-sequence transformer trained on 680,000 hours of multilingual, multitask supervision, which processes audio in 30-second windows and predicts transcript tokens together with task and timestamp markers. Its zero-shot robustness is why it became a default, and its windowing is why long-form audio needs chunking logic around it.
The two metrics you need:
, and are substitutions, deletions and insertions against a reference transcript, and is the number of reference words. WER treats every word equally, which is exactly wrong for your use case if names and numbers carry the meaning. Real-time factor below one means faster than real time; it is the number that decides whether a streaming product is feasible on a given machine.
import whisper
model = whisper.load_model("base")
result = model.transcribe(
"call.wav",
language="en",
temperature=0.0, # deterministic; avoid sampling drift
word_timestamps=True,
)
for segment in result["segments"]:
print(f"{segment['start']:.2f}-{segment['end']:.2f} {segment['text'].strip()}")For long recordings, transcribe overlapping chunks and reconcile the overlap; for streaming, use a model designed for it rather than chunking a batch model, whose 30-second context window will otherwise cut mid-sentence.
Trade-offs
| Approach | Good for | Cost | Fails when |
|---|---|---|---|
| Batch encoder–decoder (Whisper-style) | Recordings, meetings, uploads | Full-file latency, GPU memory | You need sub-second partial results |
| Streaming transducer or CTC | Live captions, voice agents | Lower accuracy, endpointing complexity | Audio is noisy or heavily accented |
| Managed speech API | Fast start, many languages | Per-minute cost, data leaves your boundary | You need on-premise processing or custom vocabulary |
| Domain fine-tune | Jargon-heavy, high-volume audio | Labelled audio, retraining, evaluation | The domain shifts or audio quality varies |
Good for
Recordings, meetings, uploads
Cost
Full-file latency, GPU memory
Fails when
You need sub-second partial results
- You need sub-second partial results
- Audio is noisy or heavily accented
- You need on-premise processing or custom vocabulary
- The domain shifts or audio quality varies
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Batch encoder–decoder (Whisper-style) is good for Recordings, meetings, uploads, costs Full-file latency, GPU memory, and fails when You need sub-second partial results. Streaming transducer or CTC is good for Live captions, voice agents, costs Lower accuracy, endpointing complexity, and fails when Audio is noisy or heavily accented. Managed speech API is good for Fast start, many languages, costs Per-minute cost, data leaves your boundary, and fails when You need on-premise processing or custom vocabulary. Domain fine-tune is good for Jargon-heavy, high-volume audio, costs Labelled audio, retraining, evaluation, and fails when The domain shifts or audio quality varies
Failure modes
- Hallucination on silence or noise. The transcript contains a sentence nobody said. Debug by checking VAD coverage for that segment, the audio's energy profile, and whether the model was run with sampling enabled.
- Entity errors. Names, drug names, tickers and numbers are consistently wrong while the rest reads well. Debug by scoring WER separately on an entity-only subset; if it diverges from overall WER, add biasing or fine-tuning rather than a bigger model.
- Chunk-boundary truncation. Words at the edge of a chunk disappear or duplicate. Debug by visualising cut points against the waveform and by testing overlap sizes on the same file.
- Timestamp drift. Word timings diverge from the audio, so captions lag. Debug by comparing predicted word boundaries with a forced alignment pass on a sample.
- Unpunctuated, unnormalised output. Downstream parsing fails on "twenty five hundred" instead of "2500". Debug by running the normaliser alone over a fixed transcript set and diffing before and after.
In production
Compute cost per audio hour, not per request: duration is the unit that matters, and RTF tells you how much compute each hour consumes. Long files should be processed asynchronously with progress reporting; interactive features should use a streaming model and accept lower accuracy for latency.
Design for human review where errors are expensive. Emit per-segment confidence, flag low-confidence spans, and let a reviewer correct them; store the corrected transcript as training and evaluation data.
Privacy is a first-class concern. Audio contains voices, which are personal data, and often contains third parties who did not consent. Redact before storing, define retention explicitly, and be careful with diarisation labels that identify speakers.
Evaluate on your own audio, stratified by channel, accent, noise level and domain. A public benchmark number tells you almost nothing about performance on telephone calls from your own customers.
Common misconceptions
People think: A low word error rate means the transcript is reliable for my task.
Actually: WER is an average over words. If your task depends on names and numbers, the errors that break it may be a small fraction of WER and a large fraction of failures.
People think: ASR transcribes what was said.
Actually: It predicts the most likely text for the acoustics. In ambiguous or silent regions it can produce fluent, confident text that was never spoken.
People think: A larger model is the fix for domain errors.
Actually: Domain errors often come from vocabulary and audio quality, not model capacity. Biasing, preprocessing and a small in-domain fine-tune frequently beat a larger general model.
Interview angle
"Build transcription for customer support calls, and make it searchable." A strong answer separates the ASR stage from the retrieval stage, chooses batch versus streaming from the latency requirement, names the metrics (entity-weighted WER, RTF), and plans for hallucination, PII and speaker attribution. Then it describes evaluation on real call audio rather than a benchmark, and the review loop for low-confidence spans. See multimodal models and the multimodal interview category.
Sources
- Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision — Whisper: 680,000 hours of weak supervision, multitask decoding, 30-second windows.
- openai/whisper — reference inference code and model card limitations.
- Hugging Face, Audio course: automatic speech recognition — CTC, seq2seq and evaluation with WER.
- Graves, Sequence Transduction with Recurrent Neural Networks — the RNN-Transducer formulation used by streaming systems.
Related
- Multimodal models — how audio and vision enter a language model.
- RAG — what you build on top of transcripts.
- Streaming and SSE — delivering partial transcripts to a client.
- Latency and TTFT — measuring responsiveness for voice.