Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 7 of 22

Audio representations

0 of 22 complete

Why it matters

Sound is a pressure signal sampled tens of thousands of times a second, and no model consumes that stream efficiently. Every audio system therefore commits to a representation: which frequencies to keep, how much time resolution to trade for frequency resolution, and whether to learn the transform. That choice propagates, because it sets the number of frames per second and therefore the latency, compute and token budget of everything after it.

Mental model

Choosing a representation is choosing a lens for a camera: a wide lens sees context and loses detail, a long lens does the reverse. The metaphor breaks because the transform is lossy in a specific, measurable way, so you can compare two representations by reconstructing the original signal and listening to what is missing.

How it works

One phrase, three representationsInteractive

The interactive waveform loads with JavaScript. The text version below describes the same frame.

Figure. A synthetic 1.6-second phrase shown as a waveform, a schematic 24-band strip drawn from those samples' energy (per-frame RMS, authored band profile), and token lanes at 100 frames/s and 50 learned units/s. The sample, frame and unit counts are arithmetic on the shown rates; no audio is recorded or played.

What to notice. The representation sets the sequence length: 25,600 samples, 160 frames or 80 units for the same phrase. All three are time-aligned on one playhead, so moving it reads the same instant in every lane. The mel strip is compact enough to model and no longer contains phase; the learned units are shorter still and tied to one encoder's preprocessing contract.

Text version

A 1.6-second synthetic phrase. The waveform runs from 0.17 to 1.42 seconds with a playhead. Below it, five lanes share the same time axis: the spoken words the, quarterly, report, is, ready; a spectrogram strip of 24 schematic bands and 160 columns; a lane of 160 frames at 10 ms hop; and 80 learned units at 50 Hz. Five steps highlight the waveform, frames, mel bands, learned units and the downstream sequence-length comparison: 25,600 samples, 160 frames, 80 units. A slider scrubs the playhead and selects the token or word under it.

  1. Decide whether the task needs phase information, which only the waveform and learned codecs preserve.
  2. Pick a window and hop size for the spectrogram, since the hop sets frames per second and therefore latency.
  3. Choose mel bins to match the frequency range of the content, not a default that assumes speech.
  4. Normalise per utterance and record the normalisation, because level differences otherwise leak into the features.
  5. If using a learned encoder, check the sample rate and preprocessing it was trained on before feeding it anything.
  6. Measure downstream accuracy and token count together, since a representation that doubles frames doubles cost.

Trade-offs

ChoiceGood forCostFails when
Raw waveformPreserves phase and transientsVery long sequencesThe model is small or data is limited
Mel spectrogramCompact and speech-matchedDiscards phase and fine frequency detailThe task is music or sound separation
Learned encoderStrong transfer with little preprocessingFixed sample rate and opaque featuresYou need to change the front end
High-resolution spectrogramDetail for music and eventsToken count and computeThe latency budget is tight
Trade-off mapInteractive

Good for

Preserves phase and transients

Cost

Very long sequences

Fails when

The model is small or data is limited

  1. The model is small or data is limited
  2. The task is music or sound separation
  3. You need to change the front end
  4. The latency budget is tight

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Raw waveform is good for Preserves phase and transients, costs Very long sequences, and fails when The model is small or data is limited. Mel spectrogram is good for Compact and speech-matched, costs Discards phase and fine frequency detail, and fails when The task is music or sound separation. Learned encoder is good for Strong transfer with little preprocessing, costs Fixed sample rate and opaque features, and fails when You need to change the front end. High-resolution spectrogram is good for Detail for music and events, costs Token count and compute, and fails when The latency budget is tight

Failure modes

  • Sample rate mismatch. The model receives audio at the wrong rate and everything sounds pitch-shifted, so accuracy collapses. Debug: assert the sample rate at ingest and log it beside every feature tensor.
  • Normalisation drift. Training used peak normalisation and serving uses RMS, so levels differ. Debug: compare feature statistics between a training batch and a live request.
  • Frame misalignment. Labels are timestamped on a different hop and training targets land on the wrong frames. Debug: overlay label boundaries on the spectrogram for a few examples.

In production

Log the sample rate, channel layout and normalisation with every request, and reject inputs that do not match the training configuration. Monitor feature statistics for drift, which catches codec changes upstream.

Common misconceptions

People think: A spectrogram is a lossless view of audio.

Actually: It discards phase and quantises frequency, and some tasks cannot be recovered from it.

People think: Higher resolution is always better.

Actually: It multiplies token count and latency, and past the content's bandwidth it adds cost without information.

Interview angle

Name the transform and its losses, then connect frame rate to latency and cost. A strong answer picks a representation from the task's need for phase and fine frequency detail. Weak answers name mel spectrograms as the default without saying what they discard or why that matters for music and event detection.

Sources