Why it matters
A model needs a finite output space, but text is open-ended. Character-level models make sequences long and dilute each token's meaning; word-level models explode the vocabulary and fail on misspellings and morphology. Subword algorithms sit between the two by learning which character sequences deserve their own id. The choice of algorithm determines how many tokens a language, a codebase or a numeric table consumes.
Mental model
A tokenizer algorithm is a compression scheme learned from a corpus: it finds repeated character patterns and gives each a short code. The metaphor breaks because the objective is not the smallest file — it is the segmentation that best supports prediction, so a slightly longer encoding can be the better tokenizer.
How it works
Paragraph window · illustrative tokens
84 input+ 24 reserve= 108 of 128
20 free
Step 1BPE · 84
Frequency merges give the paragraph 84 tokens, 20 free beside the reserve. Merge order follows counts across the corpus, so the segmentation is deterministic but not chosen for meaning.
Output reserve
24Pick a section to read its token count, its role and its state.
Figure. The content is identical in every step; the token count is not. BPE merges by frequency, WordPiece by likelihood gain and unigram by pruning, so the same paragraph needs 84, 92 or 88 tokens — and a byte-level fallback for a Cyrillic line needs 112.
What to notice. Counts are authored for one short paragraph in a 128-token window with a 24-token reserve; no tokenizer was run. The three algorithms are distinguished by their merge or pruning rule, not by a measured quality difference, and vocabulary size and training corpus are held fixed and not shown. The byte-level step shows coverage, not accuracy.
Text version
A 128-token window with a 24-token answer reserve, so 104 tokens are available. The default first step is the BPE segmentation: body 40, numbers and dates 26, closing note 18, for 84 input tokens and 20 free. The second step is WordPiece: body 44, numbers 28, closing 20, for 92 tokens and 12 free. The third step is unigram: body 42, numbers 27, closing 19, for 88 tokens and 16 free. The fourth step is a byte-level scheme on the same paragraph with a Cyrillic product line: Latin text 62 and Cyrillic line 50, for 112 tokens, 8 over the window, with the cells crossing the cap rule into the reserve. Selecting a section shows its count and what it contains.
- Normalise the corpus consistently. Unicode form, whitespace handling, case policy — and record the normaliser with the vocabulary.
- Start from a base alphabet: characters, or bytes with a reversible mapping so any input is representable.
- For BPE, count adjacent pairs across the corpus, merge the most frequent pair into a new symbol, and repeat until the target vocabulary size is reached.
- For WordPiece, score candidate merges by the likelihood gain of the resulting language model and keep the best-scoring merges.
- For unigram, start from a large candidate set, fit a unigram language model, and prune units whose removal least harms the corpus likelihood.
- Freeze the merges or scores into an artefact, version it, and use the exact same file for training and serving.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| BPE | Simple, deterministic, strong for code and mixed text | Merge order is frequency-driven, not meaning-driven | The corpus is dominated by one language or format |
| WordPiece | Good balance for natural language tasks | Training is more expensive, greedy matching at encode time | You need byte-level robustness |
| Unigram | Probabilistic segmentation, supports sampling | More complex training and inference | Determinism is required for caching |
| Large vocabulary | Shorter sequences, faster per token | Bigger embedding matrix and rarer updates per token | The corpus is small or multilingual |
Good for
Simple, deterministic, strong for code and mixed text
Cost
Merge order is frequency-driven, not meaning-driven
Fails when
The corpus is dominated by one language or format
- The corpus is dominated by one language or format
- You need byte-level robustness
- Determinism is required for caching
- The corpus is small or multilingual
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: BPE is good for Simple, deterministic, strong for code and mixed text, costs Merge order is frequency-driven, not meaning-driven, and fails when The corpus is dominated by one language or format. WordPiece is good for Good balance for natural language tasks, costs Training is more expensive, greedy matching at encode time, and fails when You need byte-level robustness. Unigram is good for Probabilistic segmentation, supports sampling, costs More complex training and inference, and fails when Determinism is required for caching. Large vocabulary is good for Shorter sequences, faster per token, costs Bigger embedding matrix and rarer updates per token, and fails when The corpus is small or multilingual
Failure modes
- Train/serve tokenizer mismatch. Training used one vocabulary file and serving loads another, so ids map to different strings. Debug: hash the tokenizer artefact and compare it against the one recorded in the checkpoint metadata.
- Unicode normalisation asymmetry. Input is normalised at training time but not at inference, so visually identical text tokenises differently. Debug: round-trip a sample of production strings through normalise-tokenise-decode and diff against the original.
- Merge rules applied in the wrong order. A hand-rolled encoder applies merges by rank incorrectly and produces ids the model never saw. Debug: compare against the reference tokenizer on a fixed corpus and locate the first divergent string.
In production
Treat the tokenizer file as part of the model artefact: hash it, ship it with the weights, and reject requests whose tokenizer version does not match. Monitor tokens per character by language and content type, because a rising ratio signals text the vocabulary handles badly.
Common misconceptions
People think: A token is roughly a word.
Actually: Tokens are learned subword units; one word can be one token, several tokens, or part of a longer token depending on frequency and the vocabulary.
People think: Any tokenizer with the same vocabulary size is equivalent.
Actually: Different algorithms segment the same string differently, which changes sequence length, cost and what the model can attend to within a fixed window.
Interview angle
A strong answer describes the algorithm as a learned segmentation with a fixed budget, contrasts merge-frequency and likelihood objectives, and explains byte fallback. It should connect tokenizer choice to sequence length, multilingual cost and artefact versioning. Weak answers say 'BPE splits words into subwords' without describing the merge procedure or its failure modes.
Sources
- Sennrich, Haddow & Birch, Neural Machine Translation of Rare Words with Subword Units — byte-pair encoding for subword segmentation.
- Kudo & Richardson, SentencePiece: A simple and language independent subword tokenizer — unigram segmentation and language-independent preprocessing.
- Hugging Face, Summary of the tokenizers — a side-by-side description of BPE, WordPiece and unigram.