Why it matters
Sound is a pressure signal sampled tens of thousands of times a second, and no model consumes that stream efficiently. Every audio system therefore commits to a representation: which frequencies to keep, how much time resolution to trade for frequency resolution, and whether to learn the transform. That choice propagates, because it sets the number of frames per second and therefore the latency, compute and token budget of everything after it.
Mental model
Choosing a representation is choosing a lens for a camera: a wide lens sees context and loses detail, a long lens does the reverse. The metaphor breaks because the transform is lossy in a specific, measurable way, so you can compare two representations by reconstructing the original signal and listening to what is missing.
How it works
The interactive waveform loads with JavaScript. The text version below describes the same frame.
Figure. A synthetic 1.6-second phrase shown as a waveform, a schematic 24-band strip drawn from those samples' energy (per-frame RMS, authored band profile), and token lanes at 100 frames/s and 50 learned units/s. The sample, frame and unit counts are arithmetic on the shown rates; no audio is recorded or played.
What to notice. The representation sets the sequence length: 25,600 samples, 160 frames or 80 units for the same phrase. All three are time-aligned on one playhead, so moving it reads the same instant in every lane. The mel strip is compact enough to model and no longer contains phase; the learned units are shorter still and tied to one encoder's preprocessing contract.
Text version
A 1.6-second synthetic phrase. The waveform runs from 0.17 to 1.42 seconds with a playhead. Below it, five lanes share the same time axis: the spoken words the, quarterly, report, is, ready; a spectrogram strip of 24 schematic bands and 160 columns; a lane of 160 frames at 10 ms hop; and 80 learned units at 50 Hz. Five steps highlight the waveform, frames, mel bands, learned units and the downstream sequence-length comparison: 25,600 samples, 160 frames, 80 units. A slider scrubs the playhead and selects the token or word under it.
- Decide whether the task needs phase information, which only the waveform and learned codecs preserve.
- Pick a window and hop size for the spectrogram, since the hop sets frames per second and therefore latency.
- Choose mel bins to match the frequency range of the content, not a default that assumes speech.
- Normalise per utterance and record the normalisation, because level differences otherwise leak into the features.
- If using a learned encoder, check the sample rate and preprocessing it was trained on before feeding it anything.
- Measure downstream accuracy and token count together, since a representation that doubles frames doubles cost.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Raw waveform | Preserves phase and transients | Very long sequences | The model is small or data is limited |
| Mel spectrogram | Compact and speech-matched | Discards phase and fine frequency detail | The task is music or sound separation |
| Learned encoder | Strong transfer with little preprocessing | Fixed sample rate and opaque features | You need to change the front end |
| High-resolution spectrogram | Detail for music and events | Token count and compute | The latency budget is tight |
Good for
Preserves phase and transients
Cost
Very long sequences
Fails when
The model is small or data is limited
- The model is small or data is limited
- The task is music or sound separation
- You need to change the front end
- The latency budget is tight
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Raw waveform is good for Preserves phase and transients, costs Very long sequences, and fails when The model is small or data is limited. Mel spectrogram is good for Compact and speech-matched, costs Discards phase and fine frequency detail, and fails when The task is music or sound separation. Learned encoder is good for Strong transfer with little preprocessing, costs Fixed sample rate and opaque features, and fails when You need to change the front end. High-resolution spectrogram is good for Detail for music and events, costs Token count and compute, and fails when The latency budget is tight
Failure modes
- Sample rate mismatch. The model receives audio at the wrong rate and everything sounds pitch-shifted, so accuracy collapses. Debug: assert the sample rate at ingest and log it beside every feature tensor.
- Normalisation drift. Training used peak normalisation and serving uses RMS, so levels differ. Debug: compare feature statistics between a training batch and a live request.
- Frame misalignment. Labels are timestamped on a different hop and training targets land on the wrong frames. Debug: overlay label boundaries on the spectrogram for a few examples.
In production
Log the sample rate, channel layout and normalisation with every request, and reject inputs that do not match the training configuration. Monitor feature statistics for drift, which catches codec changes upstream.
Common misconceptions
People think: A spectrogram is a lossless view of audio.
Actually: It discards phase and quantises frequency, and some tasks cannot be recovered from it.
People think: Higher resolution is always better.
Actually: It multiplies token count and latency, and past the content's bandwidth it adds cost without information.
Interview angle
Name the transform and its losses, then connect frame rate to latency and cost. A strong answer picks a representation from the task's need for phase and fine frequency detail. Weak answers name mel spectrograms as the default without saying what they discard or why that matters for music and event detection.
Sources
- Gong et al., AST: Audio Spectrogram Transformer — spectrogram patches as transformer input.
- Baevski et al., wav2vec 2.0 — learned representations from raw waveform.
- Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision — the log-mel front end and its fixed preprocessing assumptions.