Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 12 of 22

Speech-to-speech

0 of 22 complete

Why it matters

A cascade transcribes, reasons over text, then synthesises. That is easy to instrument and easy to break: tone, emphasis and speaker identity are discarded at the transcript, and errors compound across stages. An end-to-end model keeps audio tokens throughout, which preserves prosody but removes the text you would inspect. The choice is between observability and fidelity, and both pay a latency bill.

Mental model

A cascade is a translator taking notes and then reading them aloud: accurate content, flattened delivery. The metaphor breaks because notes can be annotated with tone and emphasis, which is what an end-to-end model learns to preserve implicitly rather than being told.

How it works

Text bottleneck or audio tokensInteractive

The interactive waveform loads with JavaScript. The text version below describes the same frame.

Figure. A synthetic 4.6-second timeline with one user turn from 0.45 to 2.85 s. Alternative reply starts are marked at 3.45 s (end-to-end, 0.60 s after the stop) and 4.13 s (cascade, 1.28 s after it); the lanes are alternatives, not two replies. Both totals are illustrative arithmetic, not measurements. The audio-token lane is drawn at 12.5 Hz for legibility.

What to notice. The cascade is a chain of inspectable stages and the transcript is a hard wall: stress and intonation do not cross it. The end-to-end path keeps audio tokens, so the reply can mirror the input's contour, and it has no transcript to audit. Choosing is choosing between observability and fidelity; turn-taking decides how fast either path feels.

Text version

A synthetic 4.6-second timeline with one user turn from 0.45 to 2.85 seconds. Lanes show the cascade transcript (book me a table for four), two prosody cues (stress on table and a rising request), a schematic end-to-end audio-token lane, and two alternative reply-start markers at 3.45 seconds for the end-to-end path and 4.13 seconds for the cascade. Steps compare the cascade stages VAD 0.08, ASR 0.30, LLM 0.50 and TTS 0.40 seconds (total 1.28) with the end-to-end encoder 0.10, model 0.30 and decoder 0.20 (total 0.60), then show the prosody lost at the text bottleneck and the different reply gaps.

  1. Decide whether the product needs the transcript, since that requirement usually forces a cascade or a parallel recogniser.
  2. Budget latency per stage and set a target for time to first audio, not only for total response time.
  3. Choose a turn policy before the model, because when to start speaking decides perceived responsiveness.
  4. Preserve speaker identity explicitly if the output must sound like the input, since most pipelines discard it.
  5. Evaluate with human ratings of naturalness and task success, and keep a transcript-based check for content accuracy.
  6. Rehearse interruptions, background noise and accented speech, which break cascades and end-to-end models differently.

Trade-offs

ChoiceGood forCostFails when
CascadeInspectable, modular, easy to logLatency and lost prosodyTone and identity matter
End-to-endNatural prosody and timingOpaque and harder to evaluateYou must audit content
Streaming in both directionsLow perceived latencyPartial-context errorsAccuracy is the priority
Turn-basedSimpler and more accurateFeels slowThe product is conversational
Trade-off mapInteractive

Good for

Inspectable, modular, easy to log

Cost

Latency and lost prosody

Fails when

Tone and identity matter

  1. Tone and identity matter
  2. You must audit content
  3. Accuracy is the priority
  4. The product is conversational

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Cascade is good for Inspectable, modular, easy to log, costs Latency and lost prosody, and fails when Tone and identity matter. End-to-end is good for Natural prosody and timing, costs Opaque and harder to evaluate, and fails when You must audit content. Streaming in both directions is good for Low perceived latency, costs Partial-context errors, and fails when Accuracy is the priority. Turn-based is good for Simpler and more accurate, costs Feels slow, and fails when The product is conversational

Failure modes

  • Compounding cascade errors. A misheard word becomes a wrong translation and then a confident sentence. Debug: log the transcript alongside the final audio and compare content on a sample.
  • Latency stacking. Each stage is acceptable and the sum is not. Debug: instrument time to first byte per stage and compute the p95 total for a real session.
  • Voice identity loss. The output voice does not match the input speaker and users notice immediately. Debug: run a speaker-similarity check on paired input and output clips.

In production

Track time to first audio and interruption handling rate per session. Keep transcripts for cascades, and for end-to-end models run an independent recogniser on the output for content auditing.

Common misconceptions

People think: End-to-end speech translation removes the need for evaluation.

Actually: It removes the transcript, so you must add one back to check content.

People think: Lower latency always means a better experience.

Actually: Speaking too early, before the user finishes, is worse than a short wait.

Interview angle

Compare the two architectures on observability, prosody and latency rather than on quality alone. A strong answer says how they would audit content in a system that produces no text. Weak answers assume an end-to-end model is strictly better and never mention how errors would be found.

Sources