Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 10 of 22

ASR

0 of 22 complete

Why ASR sits in the critical path

Speech is the cheapest input channel for users and the most expensive one to process. A voice interface, a meeting transcript, a call-centre audit or a captioned video all depend on the same conversion, and everything downstream — retrieval, summarisation, agent actions — inherits its errors.

That inheritance is asymmetric. A dropped filler word is invisible to a summary. A misheard surname, a negated instruction, or a digit in an invoice number can invert the meaning or corrupt a record. ASR quality is therefore not one number; it is a distribution of error types, and the ones that matter depend on what consumes the text.

Mental model

A transcriptionist working from sound alone, with no ability to ask "did you say fifteen or fifty?". They write the most probable sentence consistent with what they heard, using the vocabulary and phrasing they have seen most often. That is why an unusual surname becomes a common word, and why a noisy passage becomes a fluent sentence that was never spoken.

The analogy breaks on self-knowledge. A human transcriptionist marks uncertainty or leaves a gap; a neural ASR system emits its best sequence with no reliable signal that it is guessing, unless you ask for per-token scores and calibrate them. It also has no access to the conversation's context beyond what you give it, so domain hints must be supplied deliberately.

How it works

Speech to transcriptInteractive
Synthetic audio
waveform · amplitude over timeinside speech span (VAD)
schematic spectrogram · 24 bands × 72 frames · low bands at the bottomauthored VAD span 0.22–3.22 s
energyspeech span (VAD)playhead
The strip is a schematic: per-frame energy is measured from the waveform above and spread over fixed bands, so it dips with the word gaps. It is not an STFT and carries no Hz axis.
Transcript · illustrative timestamps
mean synthetic confidence 0.93
0.74 s · quarterly · synthetic confidence 0.93 — the token covering this frame.
Token 3 of 10: quarterly
Ten tokens across a 3.00 s authored speech span in a 4.00 s clip. Timestamps and confidences are authored, not decoded or calibrated; the noise is a synthetic floor, so no dB value is shown, and it lowers four token confidences while rewriting only the name. In a real pipeline, word timings come from aligning the decoded text to the audio.

Figure. A synthetic 4-second utterance: waveform, a schematic 24-band energy grid measured from the same samples, an authored VAD span (0.22–3.22 s, not detected here), and ten tokens with illustrative timestamps. No recording is used or played.

What to notice. Ten tokens is a deliberately tiny sample: one name substitution is already 10% word error rate, so the average hides nothing — the error is the whole story. In a real pipeline timestamps come from aligning the decoded text to the audio, not from the acoustic model alone.

Text version

Synthetic audio. A four-second waveform with an authored speech span from 0.22 to 3.22 seconds and a playhead. Below it, a schematic 24-band energy grid measured from the same waveform samples, brightest during speech and dimmer in the gaps between words. Below that, ten transcript tokens with illustrative timestamps: so, the, quarterly, report, goes, to, Priya, before, Friday, please. A slider scrubs the playhead and highlights the token under it. Switching noise on thickens the waveform floor, brightens every band, lowers several token confidences, and replaces Priya with the uncertain token Prya at 0.41 authored confidence.

  1. Capture and resample. Convert to mono at the model's expected sample rate (commonly 16 kHz), normalise level, and reject clipped or near-silent input early.
  2. Segment. Voice activity detection marks speech regions, trims long silences, and produces chunk boundaries. Bad segmentation is a common source of both latency and truncation.
  3. Extract features. Compute log-mel spectrogram frames over short windows with a small hop, which turns waveform into a time–frequency representation the encoder can consume.
  4. Encode acoustics. A transformer or convolution-augmented encoder turns frames into contextual hidden states. This is where noise robustness and speaker variation are absorbed.
  5. Decode. Three families dominate. CTC predicts a label per frame plus a blank, then collapses repeats — simple, non-autoregressive, strong with external language models. RNN-Transducer combines a prediction network with a joint network and streams naturally. Attention encoder–decoder models, including Whisper, generate tokens autoregressively from the encoded audio.
  6. Normalise. Detokenise, restore punctuation and casing, expand or contract numbers and dates to house style, and attach timestamps.
  7. Adapt. Apply custom vocabulary or biasing for names, products and jargon; optionally fine-tune on in-domain audio.
  8. Post-process. Diarisation, profanity masking, and redaction are separate models or rules layered on the transcript.

Whisper is the reference example for the batch case: a sequence-to-sequence transformer trained on 680,000 hours of multilingual, multitask supervision, which processes audio in 30-second windows and predicts transcript tokens together with task and timestamp markers. Its zero-shot robustness is why it became a default, and its windowing is why long-form audio needs chunking logic around it.

The two metrics you need:

WER=S+D+INRTF=processing timeaudio duration\text{WER} = \frac{S + D + I}{N} \qquad \text{RTF} = \frac{\text{processing time}}{\text{audio duration}}

SS, DD and II are substitutions, deletions and insertions against a reference transcript, and NN is the number of reference words. WER treats every word equally, which is exactly wrong for your use case if names and numbers carry the meaning. Real-time factor below one means faster than real time; it is the number that decides whether a streaming product is feasible on a given machine.

import whisper
 
model = whisper.load_model("base")
 
result = model.transcribe(
    "call.wav",
    language="en",
    temperature=0.0,        # deterministic; avoid sampling drift
    word_timestamps=True,
)
 
for segment in result["segments"]:
    print(f"{segment['start']:.2f}-{segment['end']:.2f}  {segment['text'].strip()}")

For long recordings, transcribe overlapping chunks and reconcile the overlap; for streaming, use a model designed for it rather than chunking a batch model, whose 30-second context window will otherwise cut mid-sentence.

Trade-offs

ApproachGood forCostFails when
Batch encoder–decoder (Whisper-style)Recordings, meetings, uploadsFull-file latency, GPU memoryYou need sub-second partial results
Streaming transducer or CTCLive captions, voice agentsLower accuracy, endpointing complexityAudio is noisy or heavily accented
Managed speech APIFast start, many languagesPer-minute cost, data leaves your boundaryYou need on-premise processing or custom vocabulary
Domain fine-tuneJargon-heavy, high-volume audioLabelled audio, retraining, evaluationThe domain shifts or audio quality varies
Trade-off mapInteractive

Good for

Recordings, meetings, uploads

Cost

Full-file latency, GPU memory

Fails when

You need sub-second partial results

  1. You need sub-second partial results
  2. Audio is noisy or heavily accented
  3. You need on-premise processing or custom vocabulary
  4. The domain shifts or audio quality varies

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Batch encoder–decoder (Whisper-style) is good for Recordings, meetings, uploads, costs Full-file latency, GPU memory, and fails when You need sub-second partial results. Streaming transducer or CTC is good for Live captions, voice agents, costs Lower accuracy, endpointing complexity, and fails when Audio is noisy or heavily accented. Managed speech API is good for Fast start, many languages, costs Per-minute cost, data leaves your boundary, and fails when You need on-premise processing or custom vocabulary. Domain fine-tune is good for Jargon-heavy, high-volume audio, costs Labelled audio, retraining, evaluation, and fails when The domain shifts or audio quality varies

Failure modes

  • Hallucination on silence or noise. The transcript contains a sentence nobody said. Debug by checking VAD coverage for that segment, the audio's energy profile, and whether the model was run with sampling enabled.
  • Entity errors. Names, drug names, tickers and numbers are consistently wrong while the rest reads well. Debug by scoring WER separately on an entity-only subset; if it diverges from overall WER, add biasing or fine-tuning rather than a bigger model.
  • Chunk-boundary truncation. Words at the edge of a chunk disappear or duplicate. Debug by visualising cut points against the waveform and by testing overlap sizes on the same file.
  • Timestamp drift. Word timings diverge from the audio, so captions lag. Debug by comparing predicted word boundaries with a forced alignment pass on a sample.
  • Unpunctuated, unnormalised output. Downstream parsing fails on "twenty five hundred" instead of "2500". Debug by running the normaliser alone over a fixed transcript set and diffing before and after.

In production

Compute cost per audio hour, not per request: duration is the unit that matters, and RTF tells you how much compute each hour consumes. Long files should be processed asynchronously with progress reporting; interactive features should use a streaming model and accept lower accuracy for latency.

Design for human review where errors are expensive. Emit per-segment confidence, flag low-confidence spans, and let a reviewer correct them; store the corrected transcript as training and evaluation data.

Privacy is a first-class concern. Audio contains voices, which are personal data, and often contains third parties who did not consent. Redact before storing, define retention explicitly, and be careful with diarisation labels that identify speakers.

Evaluate on your own audio, stratified by channel, accent, noise level and domain. A public benchmark number tells you almost nothing about performance on telephone calls from your own customers.

Common misconceptions

People think: A low word error rate means the transcript is reliable for my task.

Actually: WER is an average over words. If your task depends on names and numbers, the errors that break it may be a small fraction of WER and a large fraction of failures.

People think: ASR transcribes what was said.

Actually: It predicts the most likely text for the acoustics. In ambiguous or silent regions it can produce fluent, confident text that was never spoken.

People think: A larger model is the fix for domain errors.

Actually: Domain errors often come from vocabulary and audio quality, not model capacity. Biasing, preprocessing and a small in-domain fine-tune frequently beat a larger general model.

Interview angle

"Build transcription for customer support calls, and make it searchable." A strong answer separates the ASR stage from the retrieval stage, chooses batch versus streaming from the latency requirement, names the metrics (entity-weighted WER, RTF), and plans for hallucination, PII and speaker attribution. Then it describes evaluation on real call audio rather than a benchmark, and the review loop for low-confidence spans. See multimodal models and the multimodal interview category.

Sources