Why it matters
A language model's first and last layers are lookup tables indexed by integers. Text has to become integers before anything else happens, and the mapping is learned, frozen and shipped alongside the weights.
That has three consequences an engineer feels directly. Cost is billed and latency is measured in tokens, not words. Context limits are enforced in tokens, so truncation happens at token boundaries that do not align with sentences. And behaviour on numbers, code, identifiers and non-English text depends on how the tokenizer was trained — a model that splits a number into digits can count more reliably than one that treats the whole number as a rare token.
Most tokenizer problems are silent. The ids are in range, the forward pass succeeds, and the output is subtly worse because the text was segmented differently from how the model was trained.
Mental model: a fixed set of building blocks
Imagine a language made of a fixed set of LEGO bricks. Common words are one brick; rare words are assembled from smaller pieces; a word that needs a piece the set does not contain is broken into single characters or bytes. The brick set is chosen once, by frequency, and never changes.
This explains most observed behaviour. Frequent words cost one token. Rare technical terms cost several. Whitespace and punctuation are bricks of their own, so formatting is not free.
The analogy breaks because the segmentation is not designed to be linguistically meaningful. Merges are chosen by corpus statistics, so token and isation may be merged for reasons that have nothing to do with English morphology. Do not read the pieces as morphemes.
How it works
Demo window · illustrative tokens
21 input+ 8 reserve= 29 of 48
19 free
Step 1English prose
21 tokens for the sentence, leaving 19 beside the 8-token reserve. Words merge, but the identifier, amount and date do not: 12 of the 21 tokens are digits and rare strings.
Output reserve
8Pick a section to read its token count, its role and its state.
Figure. The same sentence is 21 tokens as English prose, 29 in German, 47 in Japanese and 32 as JSON. The meaning does not change; what the window can hold beside the answer does.
What to notice. The segment counts are authored to be plausible for a sentence with an identifier, an amount and a date; no tokenizer was run and none of these numbers reproduces a named model's vocabulary. The window is 48 tokens with an 8-token reserve, chosen so one sentence fills it. Character-level detail, vocabulary size and the merge procedure are omitted — this shows composition and cost, not a benchmark.
Text version
A 48-token demo window with an 8-token output reserve, so 40 input tokens are available. The default first step is the sentence 'Invoice INV-88213 totals $1,240.50 — due 2026-12-01.': prose 9 tokens, identifier 4, amount 4 and date 4, for 21 input tokens and 19 free. The second step is the German translation: prose 12, compound nouns 9, amount 4 and date 4, for 29 tokens and 11 free. The third step is the Japanese translation: prose 8, kanji and particles 21, amount 6, date 6 and Latin identifier 6, for 47 tokens, 7 over the window, with the cells crossing the cap rule into the output reserve. The fourth step is a JSON record of the same information: syntax 8, keys 7, values 11 and identifier 6, for 32 tokens and 8 free. Selecting a segment shows its count and what it holds.
- Normalise. Apply Unicode normalisation, decide how to treat whitespace, and mark word boundaries. Byte-level schemes skip most of this by operating on UTF-8 bytes.
- Start from an alphabet. Begin with characters or bytes as the base units. Byte-level starts from 256 possible byte values, which guarantees that any string is representable.
- Learn merges. Count adjacent pairs across a training corpus, merge the most frequent pair into a new unit, and repeat until the vocabulary reaches the target size. This is byte-pair encoding.
- Freeze the vocabulary. The merges and their order become part of the model artefact. Inference must apply the same merges in the same order.
- Encode new text greedily. Apply learned merges to the input, then map each resulting piece to its integer id. Unknown characters fall back to bytes or a rare-unit token, depending on the scheme.
- Add special tokens. Markers such as beginning-of-sequence, end-of-sequence, role delimiters and padding are reserved ids inserted by the chat template, not by the user's text.
- Decode. Map ids back to pieces and concatenate. Decoding is usually lossy about whitespace and is not guaranteed to reproduce the input exactly.
- Count before you send. Measure tokens with the model's own tokenizer, then apply the budget, truncation and cost calculation.
- Output
- integer ids, one per token
- Vocabulary
- fixed at training time
- Byte-level BPE
- no unknown-token case
- Cost driver
- tokens in + tokens out
The math
The merge step is a frequency maximisation. At each iteration, choose the adjacent pair with the highest count and replace every occurrence:
Two aggregate quantities are worth measuring on your own corpus. The compression ratio is characters per token, and fertility is tokens per word — both are corpus-specific, so measure rather than assume:
Cost follows from the counts:
A minimal implementation
import tiktoken
# The encoding name must match the model. This one is illustrative, not universal.
enc = tiktoken.get_encoding("cl100k_base")
text = "Invoice INV-88213 totals $1,240.50 — due 2026-12-01."
ids = enc.encode(text)
print(len(ids), "tokens") # more units than words
print([enc.decode([i]) for i in ids]) # inspect the segmentation
print(enc.decode(ids) == text) # round-trip check
# Token counts per language differ for the same meaning:
for sample in ["refund policy", "Rückerstattungsrichtlinie", "返金ポリシー"]:
print(len(enc.encode(sample)), sample)Pin the encoding with the model version and treat the name as configuration, not a constant. If the vendor changes the tokenizer for a new model, the same text yields different ids and a different bill.
Trade-offs
| Scheme | Good for | Cost | Fails when |
|---|---|---|---|
| Word-level | Simple pipelines, fixed vocabularies | Huge vocabulary, unknown-word problem | Domain has names, codes or typos |
| Character-level | No unknown tokens, small vocabulary | Long sequences, weaker long-range modelling | Context or latency budget is tight |
| BPE | General text, controllable vocabulary size | Segmentation is corpus-dependent | Corpus is dominated by one language or format |
| Byte-level BPE | Any input, including emoji and code | More tokens for non-ASCII text | You assume one token per character |
| Unigram / SentencePiece | Multilingual, no pre-tokenisation needed | Probabilistic segmentation to reproduce | You port the model without its tokenizer file |
Good for
Simple pipelines, fixed vocabularies
Cost
Huge vocabulary, unknown-word problem
Fails when
Domain has names, codes or typos
- Domain has names, codes or typos
- Context or latency budget is tight
- Corpus is dominated by one language or format
- You assume one token per character
- You port the model without its tokenizer file
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Word-level is good for Simple pipelines, fixed vocabularies, costs Huge vocabulary, unknown-word problem, and fails when Domain has names, codes or typos. Character-level is good for No unknown tokens, small vocabulary, costs Long sequences, weaker long-range modelling, and fails when Context or latency budget is tight. BPE is good for General text, controllable vocabulary size, costs Segmentation is corpus-dependent, and fails when Corpus is dominated by one language or format. Byte-level BPE is good for Any input, including emoji and code, costs More tokens for non-ASCII text, and fails when You assume one token per character. Unigram / SentencePiece is good for Multilingual, no pre-tokenisation needed, costs Probabilistic segmentation to reproduce, and fails when You port the model without its tokenizer file
Failure modes
Tokenizer mismatch. A counting library uses a different vocabulary from the model, so budgets are computed against the wrong units. Debugging step: send a probe string and compare the model's reported prompt-token count with your local count; a mismatch means the encoding is wrong.
Numbers and identifiers fragment. 1240.50 becomes several tokens, and arithmetic or exact matching degrades. Debugging step: inspect the segmentation of a representative sample of digits, IDs and URLs, and test the task's accuracy on those cases separately.
Special tokens leak from user input. A retrieved document containing a reserved marker string alters the prompt structure. Debugging step: scan untrusted text for special-token strings, and configure the tokenizer to reject them rather than encode them as ordinary text.
Truncation splits a unit. Cutting the prompt at a token limit removes half a sentence, or the closing instruction. Debugging step: measure how often truncation fires, and truncate at message or document boundaries rather than at a raw index.
In production
Measure tokens per request as a first-class metric, broken down by input and output and by language. The p95, not the mean, is what hits the context limit and what drives tail latency.
Cache on token ids where possible. Re-tokenising the same system prompt on every request is wasted CPU, and stable ids make prompt-cache prefixes easier to reason about.
Re-baseline after a model or tokenizer change. Context budgets, cost estimates, chunk sizes and truncation thresholds are all expressed in tokens; when the vocabulary changes, every one of those numbers is stale even if the application code is untouched.
Common misconceptions
People think: One token is roughly 0.75 words, so you can convert with a constant.
Actually: That ratio is a rough English-prose average. Code, JSON, identifiers, and non-Latin scripts diverge sharply, so budgets must be counted with the actual tokenizer.
People think: A bigger vocabulary is strictly better.
Actually: A larger vocabulary shortens sequences but grows the embedding and output layers, and gives each token fewer training examples. It is a trade-off, not a free win.
A third: that the tokenizer is a preprocessing detail you can swap. It defines the model's input language; a different tokenizer means a different model.
Interview angle
Tokenisation is a common screening question because it connects theory to operations: why subwords, what BPE merges, why the same sentence costs different amounts in two languages, and why a tokenizer change is a migration. Strong answers separate the algorithm from the artefact — the vocabulary is shipped with the weights — and mention the failure that follows from using a generic counting library. Read tokenizer algorithms next, and see /interview for related questions.
Sources
- Sennrich, Haddow & Birch, Neural Machine Translation of Rare Words with Subword Units — the BPE formulation for NLP: https://arxiv.org/abs/1508.07909
- Kudo & Richardson, SentencePiece: A Simple and Language Independent Subword Tokenizer — unigram segmentation without pre-tokenisation: https://arxiv.org/abs/1808.06226
- Hugging Face, Summary of the Tokenizers — side-by-side comparison of BPE, WordPiece and Unigram: https://huggingface.co/docs/transformers/tokenizer_summary
- OpenAI, tiktoken — the reference implementation used to count tokens for OpenAI models: https://github.com/openai/tiktoken