Skip to content
AI Engineer Guide.
juniorReportedcurated

Why do LLMs use subword tokenization rather than words or characters, and what does that choice cost you?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why can a model fail to count letters in a word it can otherwise use correctly?
  • The word arrives as one or two token ids, and the characters inside a token are never exposed to the model as separate items it can iterate over.
  • So the model answers from patterns learned about that token, and a word it has rarely seen has no reliable letter-level pattern to fall back on.
  • Making it emit the letters as separate tokens puts them in the context, which sometimes helps and sometimes produces a confident wrong spelling.
  • Byte-level fallback guarantees the text encodes, but for non-Latin scripts one character can take several tokens, so character-to-token alignment is lost and letter-level reasoning is harder.
How does tokenizer choice affect multilingual pricing?
  • Pricing is per token, so a language with high fertility pays more for the same content: two to four times the tokens means two to four times the input bill.
  • The same effect consumes context, so a prompt that fits in one language truncates in another at the same character count.
  • Track tokens per request and truncation rate by language, because a shift in either is usually a tokenizer or input-format problem rather than a model regression.
  • A larger context window raises the ceiling but does not change the price per token.

Red flags

  • Says characters are simply too hard for neural networks
  • Cannot connect tokenization to a concrete product failure

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals