Why it matters
Writing and speech encode different information. Text has no pauses, stress or intonation, so a synthesiser must infer them, and that inference is where most unnaturalness comes from. Add the requirement to say a part number correctly, to sound like a particular speaker, or to start playing within a fraction of a second, and the problem becomes several coupled decisions rather than one model choice.
Mental model
Text-to-speech is sight-reading: the notes are fixed and the performance is inferred, including where to breathe and which word to stress. The metaphor breaks because a sight-reader understands the music, while a synthesiser infers prosody from statistical patterns and has no idea what the sentence means.
How it works
The interactive waveform loads with JavaScript. The text version below describes the same frame.
Figure. A synthetic 2.6-second utterance for “Your appointment is at 4:15 on Friday.” The normaliser expands 4:15 to “four fifteen”; 130 acoustic tokens at 50 Hz are arithmetic on the shown duration. Timings, prosody marks and the 0.32 s first-audio figure are authored, and no audio is played.
What to notice. Normalisation happens before the acoustic model, so a wrong expansion sounds fluent and says the wrong thing. Prosody — stress and pauses — is inferred from text, not marked in it; the vocoder only renders the contour it is given. Streaming synthesis buys time to first audio with lookahead, and a domain part number can fail while every sample is clean.
Text version
A synthetic 2.6-second spoken sentence. Lanes share a time axis: input text including 4:15, the spoken words Your appointment is at four fifteen on Friday, 130 acoustic tokens at 50 Hz, two authored prosody spans for stress and pause, the 0.32 second silence before first audio, and a failure lane where A-12 is read as A one two. Steps walk normalisation, token prediction, vocoding, the first-audio trade, inferred prosody and the part-number failure. A slider scrubs the playhead.
- Normalise text before synthesis: expand numbers, currency, dates and abbreviations, and mark language switches.
- Choose the acoustic representation, either a mel spectrogram with a vocoder or discrete tokens with a decoder.
- Decide the voice strategy: a fixed voice, a cloned voice, or a small set of selectable speakers.
- Set the streaming strategy, since sentence-level lookahead improves prosody at the cost of time to first audio.
- Evaluate with human listening tests on the actual domain vocabulary, because proper nouns are where it breaks.
- Keep a fallback voice or a recorded prompt for failures, since synthesis errors are audible and immediate.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Non-streaming synthesis | Best prosody | Time to first audio | The product is conversational |
| Streaming synthesis | Fast start | Flatter prosody and revision risk | Long sentences need natural intonation |
| Voice cloning | Personalisation | Consent, safety and data handling | You cannot verify consent |
| Fixed voice | Predictable and safe | No personalisation | The brand requires a specific voice |
Good for
Best prosody
Cost
Time to first audio
Fails when
The product is conversational
- The product is conversational
- Long sentences need natural intonation
- You cannot verify consent
- The brand requires a specific voice
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Non-streaming synthesis is good for Best prosody, costs Time to first audio, and fails when The product is conversational. Streaming synthesis is good for Fast start, costs Flatter prosody and revision risk, and fails when Long sentences need natural intonation. Voice cloning is good for Personalisation, costs Consent, safety and data handling, and fails when You cannot verify consent. Fixed voice is good for Predictable and safe, costs No personalisation, and fails when The brand requires a specific voice
Failure modes
- Text normalisation error. A date or part number is read digit by digit. Debug: run a corpus of domain strings through the normaliser and read the output text before synthesis.
- Prosody collapse. Long sentences are read with uniform stress and sound robotic. Debug: compare pause distribution and pitch variance against human recordings of the same text.
- Vocoder artefact. Buzz or metallic noise appears on sustained vowels. Debug: synthesise a sustained vowel and inspect the spectrogram for periodic artefacts.
In production
Monitor time to first audio and failure rate per voice, and cache audio for fixed prompts. Run a listening check on new domain vocabulary before it reaches users.
Common misconceptions
People think: TTS is solved for any text.
Actually: It degrades on names, mixed languages and unusual punctuation, which is most real content.
People think: A better vocoder fixes naturalness.
Actually: Prosody prediction from text is usually the larger source of unnaturalness.
Interview angle
Separate the text front end, the acoustic model and the vocoder, and say which one produces the failure you hear. A strong answer mentions normalisation and streaming lookahead. Weak answers attribute every artefact to the vocoder or assume naturalness and controllability improve together.
Sources
- Shen et al., Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions — Tacotron 2 and the acoustic model plus vocoder split.
- Wang et al., Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers — VALL-E and token-based zero-shot synthesis.
- Kong et al., HiFi-GAN — the vocoder stage and its artefacts.