Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 9 of 33

Tokenisation

0 of 33 complete

Why it matters

A language model's first and last layers are lookup tables indexed by integers. Text has to become integers before anything else happens, and the mapping is learned, frozen and shipped alongside the weights.

That has three consequences an engineer feels directly. Cost is billed and latency is measured in tokens, not words. Context limits are enforced in tokens, so truncation happens at token boundaries that do not align with sentences. And behaviour on numbers, code, identifiers and non-English text depends on how the tokenizer was trained — a model that splits a number into digits can count more reliably than one that treats the whole number as a rare token.

Most tokenizer problems are silent. The ids are in range, the forward pass succeeds, and the output is subtly worse because the text was segmented differently from how the model was trained.

Mental model: a fixed set of building blocks

Imagine a language made of a fixed set of LEGO bricks. Common words are one brick; rare words are assembled from smaller pieces; a word that needs a piece the set does not contain is broken into single characters or bytes. The brick set is chosen once, by frequency, and never changes.

This explains most observed behaviour. Frequent words cost one token. Rare technical terms cost several. Whitespace and punctuation are bricks of their own, so formatting is not free.

The analogy breaks because the segmentation is not designed to be linguistically meaningful. Merges are chosen by corpus statistics, so token and isation may be merged for reasons that have nothing to do with English morphology. Do not read the pieces as morphemes.

How it works

One sentence, four token countsInteractive

Demo window · illustrative tokens

21 input+ 8 reserve= 29 of 48

19 free

Step 1/4 · English prose

Step 1English prose

21 tokens for the sentence, leaving 19 beside the 8-token reserve. Words merge, but the identifier, amount and date do not: 12 of the 21 tokens are digits and rare strings.

reserve 821 tokens

Output reserve

8
output reserveBar length is each block's token share of the window.

Pick a section to read its token count, its role and its state.

Figure. The same sentence is 21 tokens as English prose, 29 in German, 47 in Japanese and 32 as JSON. The meaning does not change; what the window can hold beside the answer does.

What to notice. The segment counts are authored to be plausible for a sentence with an identifier, an amount and a date; no tokenizer was run and none of these numbers reproduces a named model's vocabulary. The window is 48 tokens with an 8-token reserve, chosen so one sentence fills it. Character-level detail, vocabulary size and the merge procedure are omitted — this shows composition and cost, not a benchmark.

Text version

A 48-token demo window with an 8-token output reserve, so 40 input tokens are available. The default first step is the sentence 'Invoice INV-88213 totals $1,240.50 — due 2026-12-01.': prose 9 tokens, identifier 4, amount 4 and date 4, for 21 input tokens and 19 free. The second step is the German translation: prose 12, compound nouns 9, amount 4 and date 4, for 29 tokens and 11 free. The third step is the Japanese translation: prose 8, kanji and particles 21, amount 6, date 6 and Latin identifier 6, for 47 tokens, 7 over the window, with the cells crossing the cap rule into the output reserve. The fourth step is a JSON record of the same information: syntax 8, keys 7, values 11 and identifier 6, for 32 tokens and 8 free. Selecting a segment shows its count and what it holds.

  1. Normalise. Apply Unicode normalisation, decide how to treat whitespace, and mark word boundaries. Byte-level schemes skip most of this by operating on UTF-8 bytes.
  2. Start from an alphabet. Begin with characters or bytes as the base units. Byte-level starts from 256 possible byte values, which guarantees that any string is representable.
  3. Learn merges. Count adjacent pairs across a training corpus, merge the most frequent pair into a new unit, and repeat until the vocabulary reaches the target size. This is byte-pair encoding.
  4. Freeze the vocabulary. The merges and their order become part of the model artefact. Inference must apply the same merges in the same order.
  5. Encode new text greedily. Apply learned merges to the input, then map each resulting piece to its integer id. Unknown characters fall back to bytes or a rare-unit token, depending on the scheme.
  6. Add special tokens. Markers such as beginning-of-sequence, end-of-sequence, role delimiters and padding are reserved ids inserted by the chat template, not by the user's text.
  7. Decode. Map ids back to pieces and concatenate. Decoding is usually lossy about whitespace and is not guaranteed to reproduce the input exactly.
  8. Count before you send. Measure tokens with the model's own tokenizer, then apply the budget, truncation and cost calculation.
Output
integer ids, one per token
Vocabulary
fixed at training time
Byte-level BPE
no unknown-token case
Cost driver
tokens in + tokens out

The math

The merge step is a frequency maximisation. At each iteration, choose the adjacent pair with the highest count and replace every occurrence:

(a,b)=arg⁡max⁡(x,y)count(x,y),V←V∪{ab}.(a, b) = \arg\max_{(x, y)} \text{count}(x, y), \qquad V \leftarrow V \cup \{ab\}.

Two aggregate quantities are worth measuring on your own corpus. The compression ratio is characters per token, and fertility is tokens per word — both are corpus-specific, so measure rather than assume:

compression=characterstokens,fertility=tokenswords.\text{compression} = \frac{\text{characters}}{\text{tokens}}, \qquad \text{fertility} = \frac{\text{tokens}}{\text{words}}.

Cost follows from the counts:

cost=nin⋅pin+nout⋅pout.\text{cost} = n_{\text{in}} \cdot p_{\text{in}} + n_{\text{out}} \cdot p_{\text{out}}.

A minimal implementation

import tiktoken
 
# The encoding name must match the model. This one is illustrative, not universal.
enc = tiktoken.get_encoding("cl100k_base")
 
text = "Invoice INV-88213 totals $1,240.50 — due 2026-12-01."
ids = enc.encode(text)
 
print(len(ids), "tokens")                    # more units than words
print([enc.decode([i]) for i in ids])        # inspect the segmentation
print(enc.decode(ids) == text)               # round-trip check
 
# Token counts per language differ for the same meaning:
for sample in ["refund policy", "Rückerstattungsrichtlinie", "返金ポリシー"]:
    print(len(enc.encode(sample)), sample)

Pin the encoding with the model version and treat the name as configuration, not a constant. If the vendor changes the tokenizer for a new model, the same text yields different ids and a different bill.

Trade-offs

SchemeGood forCostFails when
Word-levelSimple pipelines, fixed vocabulariesHuge vocabulary, unknown-word problemDomain has names, codes or typos
Character-levelNo unknown tokens, small vocabularyLong sequences, weaker long-range modellingContext or latency budget is tight
BPEGeneral text, controllable vocabulary sizeSegmentation is corpus-dependentCorpus is dominated by one language or format
Byte-level BPEAny input, including emoji and codeMore tokens for non-ASCII textYou assume one token per character
Unigram / SentencePieceMultilingual, no pre-tokenisation neededProbabilistic segmentation to reproduceYou port the model without its tokenizer file
Trade-off mapInteractive

Good for

Simple pipelines, fixed vocabularies

Cost

Huge vocabulary, unknown-word problem

Fails when

Domain has names, codes or typos

  1. Domain has names, codes or typos
  2. Context or latency budget is tight
  3. Corpus is dominated by one language or format
  4. You assume one token per character
  5. You port the model without its tokenizer file

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Word-level is good for Simple pipelines, fixed vocabularies, costs Huge vocabulary, unknown-word problem, and fails when Domain has names, codes or typos. Character-level is good for No unknown tokens, small vocabulary, costs Long sequences, weaker long-range modelling, and fails when Context or latency budget is tight. BPE is good for General text, controllable vocabulary size, costs Segmentation is corpus-dependent, and fails when Corpus is dominated by one language or format. Byte-level BPE is good for Any input, including emoji and code, costs More tokens for non-ASCII text, and fails when You assume one token per character. Unigram / SentencePiece is good for Multilingual, no pre-tokenisation needed, costs Probabilistic segmentation to reproduce, and fails when You port the model without its tokenizer file

Failure modes

Tokenizer mismatch. A counting library uses a different vocabulary from the model, so budgets are computed against the wrong units. Debugging step: send a probe string and compare the model's reported prompt-token count with your local count; a mismatch means the encoding is wrong.

Numbers and identifiers fragment. 1240.50 becomes several tokens, and arithmetic or exact matching degrades. Debugging step: inspect the segmentation of a representative sample of digits, IDs and URLs, and test the task's accuracy on those cases separately.

Special tokens leak from user input. A retrieved document containing a reserved marker string alters the prompt structure. Debugging step: scan untrusted text for special-token strings, and configure the tokenizer to reject them rather than encode them as ordinary text.

Truncation splits a unit. Cutting the prompt at a token limit removes half a sentence, or the closing instruction. Debugging step: measure how often truncation fires, and truncate at message or document boundaries rather than at a raw index.

In production

Measure tokens per request as a first-class metric, broken down by input and output and by language. The p95, not the mean, is what hits the context limit and what drives tail latency.

Cache on token ids where possible. Re-tokenising the same system prompt on every request is wasted CPU, and stable ids make prompt-cache prefixes easier to reason about.

Re-baseline after a model or tokenizer change. Context budgets, cost estimates, chunk sizes and truncation thresholds are all expressed in tokens; when the vocabulary changes, every one of those numbers is stale even if the application code is untouched.

Common misconceptions

People think: One token is roughly 0.75 words, so you can convert with a constant.

Actually: That ratio is a rough English-prose average. Code, JSON, identifiers, and non-Latin scripts diverge sharply, so budgets must be counted with the actual tokenizer.

People think: A bigger vocabulary is strictly better.

Actually: A larger vocabulary shortens sequences but grows the embedding and output layers, and gives each token fewer training examples. It is a trade-off, not a free win.

A third: that the tokenizer is a preprocessing detail you can swap. It defines the model's input language; a different tokenizer means a different model.

Interview angle

Tokenisation is a common screening question because it connects theory to operations: why subwords, what BPE merges, why the same sentence costs different amounts in two languages, and why a tokenizer change is a migration. Strong answers separate the algorithm from the artefact — the vocabulary is shipped with the weights — and mention the failure that follows from using a generic counting library. Read tokenizer algorithms next, and see /interview for related questions.

Sources