Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 12 of 33

Subword tokenisation

0 of 33 complete

Why it matters

Word-level vocabularies cannot handle inflection, misspellings, identifiers or new terms, and character-level models make sequences too long to attend over. Subword tokenisation resolves the tension by learning frequent fragments, so an unseen word still decomposes into known pieces. The cost is that sequence length becomes a property of the tokenizer and the text's language rather than of meaning, which directly affects cost, latency and what fits in context.

Mental model

Subword tokenisation is cutting fabric along the seams that appeared most often: common patterns become single pieces, unfamiliar cloth is cut into smaller strips. The metaphor breaks because the seams are chosen by frequency in a training corpus, not by any notion of where a word's meaning actually divides.

How it works

Fertility inside one windowInteractive

Message window · illustrative tokens

156 input+ 80 reserve= 236 of 400

164 free

Step 1/4 · English · 156

Step 1English · 156

156 tokens for a 112-word message, or 1.4 tokens per word. Two thirds of it is common words the vocabulary holds whole, so most of the window is still free.

reserve 80156 tokens

Output reserve

80
output reserveBar length is each block's token share of the window.

Pick a section to read its token count, its role and its state.

Figure. The same 112-word message is 156 tokens in English, 230 in German and 348 in Japanese. Fertility — tokens per word — is a property of the tokenizer and the language, so an identical context buys less text in some languages.

What to notice. Counts are authored for a 400-token window with an 80-token reserve and do not reproduce any named tokenizer. The message, its translations and the byte-level probe are illustrative; no text was tokenized and no model was called. The figure omits vocabulary size, the training mix and per-token pricing — it shows how much of a fixed window a language consumes.

Text version

A 400-token window with an 80-token output reserve, so 320 tokens are available. The default first step is an English message: common words 96, technical terms 44 and punctuation 16, for 156 input tokens and 164 free. The second step is the same message in German: common words 104, compounds 108 and punctuation 18, for 230 tokens and 90 free. The third step is Japanese: kana and common words 78, kanji 214, byte fallback 40 and punctuation 16, for 348 tokens — 28 over the window, with the input crossing the reserve rule. A note says that is 3.1 tokens per word against 1.4 in English. The fourth step is a byte-level probe: eight emoji cost 32 tokens and a surrounding line 20, for 52 tokens, showing that the fallback always encodes but at several tokens per glyph. Selecting a segment shows its count and role.

  1. Train the tokenizer on a corpus that resembles production text, including code, numbers and every language you intend to serve.
  2. Inspect segmentation of representative strings. identifiers, URLs, dates, accented text — rather than trusting aggregate statistics.
  3. Measure fertility per language and per content type on held-out text to see where the context budget is being consumed.
  4. Add pre-tokenisation rules for whitespace and punctuation that match how your inputs actually arrive.
  5. Keep a byte-level fallback so unknown characters encode instead of raising or collapsing to a single unknown id.
  6. Freeze the tokenizer and store it with the model, then use the same version in evaluation and serving.

Trade-offs

ChoiceGood forCostFails when
Aggressive mergingShort sequences for common textPoor coverage of rare words and morphologyInputs are identifiers, codes or low-resource languages
Character-level unitsFull coverage, tiny vocabularyLong sequences and weaker per-token meaningContext or latency is constrained
Byte-level BPENever fails to encode, handles any scriptMulti-byte characters cost several tokensText is dominated by non-Latin scripts
Domain-specific tokenizerEfficient on code or scientific textPoor transfer to general languageThe product serves mixed traffic
Trade-off mapInteractive

Good for

Short sequences for common text

Cost

Poor coverage of rare words and morphology

Fails when

Inputs are identifiers, codes or low-resource languages

  1. Inputs are identifiers, codes or low-resource languages
  2. Context or latency is constrained
  3. Text is dominated by non-Latin scripts
  4. The product serves mixed traffic

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Aggressive merging is good for Short sequences for common text, costs Poor coverage of rare words and morphology, and fails when Inputs are identifiers, codes or low-resource languages. Character-level units is good for Full coverage, tiny vocabulary, costs Long sequences and weaker per-token meaning, and fails when Context or latency is constrained. Byte-level BPE is good for Never fails to encode, handles any script, costs Multi-byte characters cost several tokens, and fails when Text is dominated by non-Latin scripts. Domain-specific tokenizer is good for Efficient on code or scientific text, costs Poor transfer to general language, and fails when The product serves mixed traffic

Failure modes

  • Fertility blow-up. One language or format consumes several times more tokens than expected, so prompts truncate and costs rise. Debug: compute tokens per word by language on a live sample and compare against the training mix.
  • Whitespace-sensitive splits. Leading spaces are absorbed into tokens inconsistently, so the same phrase encodes differently at different positions. Debug: tokenise a phrase at the start, middle and end of a sentence and compare ids.
  • Glitch tokens. Rare vocabulary entries were barely trained and produce erratic behaviour when they appear. Debug: scan the vocabulary for ids whose training frequency is near zero and check model outputs on them.

In production

Track tokens per request and truncation rate by language and content type; a shift in either is usually a tokenizer or input-format problem, not a model regression. Budget context in tokens, never in words, and test with the longest realistic input rather than the average.

Common misconceptions

People think: Tokenisation is language-neutral.

Actually: Fertility varies widely: the same sentence can need far more tokens in one language or script, which changes cost and how much content fits in the window.

People think: A larger context window removes tokenisation concerns.

Actually: It raises the ceiling but not the price per token, and truncation still happens at the boundary of whatever budget you can afford.

Interview angle

A strong answer explains why subwords exist, then quantifies the consequence: fertility, context budget and cost per language. It should mention byte fallback and tokenizer versioning. Weak answers say subwords handle unknown words and stop, missing the operational effect on truncation and multilingual spend.

Sources