juniorReportedcurated
Why do LLMs use subword tokenization rather than words or characters, and what does that choice cost you?
30-second answer
- Word-level vocabularies cannot cover names, typos, morphology and code without an unbounded size or a lossy unknown token, while character-level models produce sequences four to five times longer.
- Longer sequences are expensive because attention cost grows quadratically, so character-level models pay in compute and in how much text fits in context.
- Subword tokenisation learns frequent chunks, keeps the vocabulary around 32k–256k, and with a byte-level alphabet can represent any string.
- The cost is that the model never sees characters, so letter counting, spelling and character-level edits fail because the unit it reasons over is a fragment.
- Fertility, tokens per word, varies by language, so the same content can cost two to four times more tokens in Japanese or Arabic and inflate both price and context use.
Likely follow-ups
Open a question to rehearse the answer.
Why can a model fail to count letters in a word it can otherwise use correctly?
- The word arrives as one or two token ids, and the characters inside a token are never exposed to the model as separate items it can iterate over.
- So the model answers from patterns learned about that token, and a word it has rarely seen has no reliable letter-level pattern to fall back on.
- Making it emit the letters as separate tokens puts them in the context, which sometimes helps and sometimes produces a confident wrong spelling.
- Byte-level fallback guarantees the text encodes, but for non-Latin scripts one character can take several tokens, so character-to-token alignment is lost and letter-level reasoning is harder.
How does tokenizer choice affect multilingual pricing?
- Pricing is per token, so a language with high fertility pays more for the same content: two to four times the tokens means two to four times the input bill.
- The same effect consumes context, so a prompt that fits in one language truncates in another at the same character count.
- Track tokens per request and truncation rate by language, because a shift in either is usually a tokenizer or input-format problem rather than a model regression.
- A larger context window raises the ceiling but does not change the price per token.
Red flags
- Says characters are simply too hard for neural networks
- Cannot connect tokenization to a concrete product failure
Concepts this tests
Tags
Source: github.com