Why it matters
Word-level vocabularies cannot handle inflection, misspellings, identifiers or new terms, and character-level models make sequences too long to attend over. Subword tokenisation resolves the tension by learning frequent fragments, so an unseen word still decomposes into known pieces. The cost is that sequence length becomes a property of the tokenizer and the text's language rather than of meaning, which directly affects cost, latency and what fits in context.
Mental model
Subword tokenisation is cutting fabric along the seams that appeared most often: common patterns become single pieces, unfamiliar cloth is cut into smaller strips. The metaphor breaks because the seams are chosen by frequency in a training corpus, not by any notion of where a word's meaning actually divides.
How it works
Message window · illustrative tokens
156 input+ 80 reserve= 236 of 400
164 free
Step 1English · 156
156 tokens for a 112-word message, or 1.4 tokens per word. Two thirds of it is common words the vocabulary holds whole, so most of the window is still free.
Output reserve
80Pick a section to read its token count, its role and its state.
Figure. The same 112-word message is 156 tokens in English, 230 in German and 348 in Japanese. Fertility — tokens per word — is a property of the tokenizer and the language, so an identical context buys less text in some languages.
What to notice. Counts are authored for a 400-token window with an 80-token reserve and do not reproduce any named tokenizer. The message, its translations and the byte-level probe are illustrative; no text was tokenized and no model was called. The figure omits vocabulary size, the training mix and per-token pricing — it shows how much of a fixed window a language consumes.
Text version
A 400-token window with an 80-token output reserve, so 320 tokens are available. The default first step is an English message: common words 96, technical terms 44 and punctuation 16, for 156 input tokens and 164 free. The second step is the same message in German: common words 104, compounds 108 and punctuation 18, for 230 tokens and 90 free. The third step is Japanese: kana and common words 78, kanji 214, byte fallback 40 and punctuation 16, for 348 tokens — 28 over the window, with the input crossing the reserve rule. A note says that is 3.1 tokens per word against 1.4 in English. The fourth step is a byte-level probe: eight emoji cost 32 tokens and a surrounding line 20, for 52 tokens, showing that the fallback always encodes but at several tokens per glyph. Selecting a segment shows its count and role.
- Train the tokenizer on a corpus that resembles production text, including code, numbers and every language you intend to serve.
- Inspect segmentation of representative strings. identifiers, URLs, dates, accented text — rather than trusting aggregate statistics.
- Measure fertility per language and per content type on held-out text to see where the context budget is being consumed.
- Add pre-tokenisation rules for whitespace and punctuation that match how your inputs actually arrive.
- Keep a byte-level fallback so unknown characters encode instead of raising or collapsing to a single unknown id.
- Freeze the tokenizer and store it with the model, then use the same version in evaluation and serving.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Aggressive merging | Short sequences for common text | Poor coverage of rare words and morphology | Inputs are identifiers, codes or low-resource languages |
| Character-level units | Full coverage, tiny vocabulary | Long sequences and weaker per-token meaning | Context or latency is constrained |
| Byte-level BPE | Never fails to encode, handles any script | Multi-byte characters cost several tokens | Text is dominated by non-Latin scripts |
| Domain-specific tokenizer | Efficient on code or scientific text | Poor transfer to general language | The product serves mixed traffic |
Good for
Short sequences for common text
Cost
Poor coverage of rare words and morphology
Fails when
Inputs are identifiers, codes or low-resource languages
- Inputs are identifiers, codes or low-resource languages
- Context or latency is constrained
- Text is dominated by non-Latin scripts
- The product serves mixed traffic
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Aggressive merging is good for Short sequences for common text, costs Poor coverage of rare words and morphology, and fails when Inputs are identifiers, codes or low-resource languages. Character-level units is good for Full coverage, tiny vocabulary, costs Long sequences and weaker per-token meaning, and fails when Context or latency is constrained. Byte-level BPE is good for Never fails to encode, handles any script, costs Multi-byte characters cost several tokens, and fails when Text is dominated by non-Latin scripts. Domain-specific tokenizer is good for Efficient on code or scientific text, costs Poor transfer to general language, and fails when The product serves mixed traffic
Failure modes
- Fertility blow-up. One language or format consumes several times more tokens than expected, so prompts truncate and costs rise. Debug: compute tokens per word by language on a live sample and compare against the training mix.
- Whitespace-sensitive splits. Leading spaces are absorbed into tokens inconsistently, so the same phrase encodes differently at different positions. Debug: tokenise a phrase at the start, middle and end of a sentence and compare ids.
- Glitch tokens. Rare vocabulary entries were barely trained and produce erratic behaviour when they appear. Debug: scan the vocabulary for ids whose training frequency is near zero and check model outputs on them.
In production
Track tokens per request and truncation rate by language and content type; a shift in either is usually a tokenizer or input-format problem, not a model regression. Budget context in tokens, never in words, and test with the longest realistic input rather than the average.
Common misconceptions
People think: Tokenisation is language-neutral.
Actually: Fertility varies widely: the same sentence can need far more tokens in one language or script, which changes cost and how much content fits in the window.
People think: A larger context window removes tokenisation concerns.
Actually: It raises the ceiling but not the price per token, and truncation still happens at the boundary of whatever budget you can afford.
Interview angle
A strong answer explains why subwords exist, then quantifies the consequence: fertility, context budget and cost per language. It should mention byte fallback and tokenizer versioning. Weak answers say subwords handle unknown words and stop, missing the operational effect on truncation and multilingual spend.
Sources
- Sennrich, Haddow & Birch, Neural Machine Translation of Rare Words with Subword Units — the subword approach and its handling of rare words.
- Radford et al., Language Models are Unsupervised Multitask Learners — byte-level BPE and reversible encoding.
- Kudo & Richardson, SentencePiece — language-independent tokenisation from raw text.