Why it matters
Tokens are the unit in which every LLM constraint is expressed. The context window is a token budget. Pricing is per input and output token. Throughput is tokens per second. Truncation happens at a token boundary, which is why a cut prompt can end mid-word or mid-JSON.
Misjudging tokens produces a specific class of bug: a prompt that passes in testing and overflows in production once a user pastes more text, a JSON response that is valid in staging and truncated in the field, or a monthly bill that is double the estimate because the content is not English prose. Counting tokens with the wrong tokeniser is a silent error: the count looks plausible and is simply not the number the provider will charge.
Mental model
A compression dictionary. The tokeniser learns frequent character sequences and assigns each one an ID; common words become one entry, rarer words split into pieces.
The analogy breaks because the dictionary is not optimised for meaning or for human readability. It is optimised for compression on a training corpus. That is why " the" with a leading space can be a single token, why a date can split into several, and why a word that is common in one language may cost multiple tokens in another.
How it works
- 1Normalise the text and split it into candidate pieces at whitespace and punctuation boundaries.
- 2Apply the vocabulary's merge rules to combine frequent adjacent pieces into larger units.
- 3Map each resulting piece to an integer ID from the fixed vocabulary.
- 4Insert special tokens that mark roles, boundaries, or tool calls.
- 5Send the ID sequence to the model; detokenise the output IDs back to text.
Byte-pair encoding is the common recipe. Start with a base vocabulary of characters or bytes, count adjacent pairs in a corpus, and repeatedly merge the most frequent pair into a new symbol. Each merge is one vocabulary entry. After enough merges, frequent words become single tokens and rare words remain compositions of common pieces. SentencePiece-style unigram tokenisers take a different route — they fit a probabilistic model over subword candidates and prune the vocabulary — but the interface is the same: text in, IDs out.
The vocabulary is fixed when the model is trained. A model cannot read a token it has never seen, so genuinely novel strings are decomposed into bytes or characters. This is why tokenisers handle arbitrary Unicode without an unknown-token failure, and why unusual scripts and emoji can be expensive: each may be several tokens.
Detokenisation is not always the exact inverse of tokenisation. Whitespace handling differs by tokeniser, and a token may encode a leading space. Round-tripping text through tokens and back is usually safe for content, but do not rely on it for byte-exact reproduction.
Quick facts
- Unit
- Subword ID
- Vocabulary
- Fixed at training time
- Characters per token
- Varies by tokenizer and language
- Code and JSON
- Often more tokens per character
- Pricing unit
- Input and output tokens
- Truncation
- Always at a token boundary
OpenAI documents a rough English heuristic of about four characters per token. Treat it as a planning default only, never as a billing estimate.
The arithmetic that matters
For a request with input tokens and output tokens:
and the space left for output is:
where is the context window and is headroom for tokeniser drift, tool schemas, and provider-side framing tokens you did not count. A margin of zero is a truncation bug waiting for a longer input. Output tokens usually cost more than input tokens, so a cheaper prompt does not always mean a cheaper request: a verbose answer can dominate the bill.
Counting before you send
import tiktoken
def count_tokens(text: str, model_name: str) -> int:
# The encoding is tied to the model family; never reuse a count across models.
enc = tiktoken.encoding_for_model(model_name)
return len(enc.encode(text))
def fits(text: str, model_name: str, window: int, reserve: int = 512) -> bool:
return count_tokens(text, model_name) + reserve <= windowThe exact function names differ across tokeniser libraries, but the contract is the same: obtain the encoding for the target model, encode, and count. For a message list, count the serialised messages plus the per-message overhead the provider adds; a rough count of the visible text understates the request.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Byte-level BPE | Arbitrary Unicode, code, no unknown tokens | Longer sequences for non-English text | You need short sequences in many scripts |
| SentencePiece unigram | Multilingual coverage, language-agnostic pipelines | Model-specific behaviour, less transparent | You assume counts transfer between models |
| Word-level vocabulary | Short sequences for common words | Huge vocabulary, unknown-token failures | Input is code, names, or misspellings |
| Character-level | Tiny vocabulary, robust to novelty | Very long sequences, slow generation | Context or latency is constrained |
Good for
Arbitrary Unicode, code, no unknown tokens
Cost
Longer sequences for non-English text
Fails when
You need short sequences in many scripts
- You need short sequences in many scripts
- You assume counts transfer between models
- Input is code, names, or misspellings
- Context or latency is constrained
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Byte-level BPE is good for Arbitrary Unicode, code, no unknown tokens, costs Longer sequences for non-English text, and fails when You need short sequences in many scripts. SentencePiece unigram is good for Multilingual coverage, language-agnostic pipelines, costs Model-specific behaviour, less transparent, and fails when You assume counts transfer between models. Word-level vocabulary is good for Short sequences for common words, costs Huge vocabulary, unknown-token failures, and fails when Input is code, names, or misspellings. Character-level is good for Tiny vocabulary, robust to novelty, costs Very long sequences, slow generation, and fails when Context or latency is constrained
Failure modes
Counting with the wrong tokeniser. The estimate is close enough to pass review and wrong at the limit. Debug by logging the provider's reported usage alongside your local count; a persistent gap means the encoding or the message overhead is wrong.
Truncating serialised output. A response is cut at the output cap, leaving invalid JSON. Debug by checking the finish reason, not the text. If it is a length stop, raise the cap or shorten the prompt; do not add a repair prompt before fixing the budget.
Numeric and identifier errors. Long numbers, IDs and dates can split into pieces that obscure place-value structure. Debug by testing the exact formats you care about; if the model mangles them, extract them with code or a tool rather than relying on generation.
Cost drift without traffic growth. Average prompt size grows because a template gained examples or a retrieved document got longer. Debug by tracking tokens per request by endpoint and by prompt version, not just total spend.
Symptom
The estimate is close enough to pass review and wrong at the limit.
Debug step
Logging the provider's reported usage alongside your local count; a persistent gap means the encoding or the message overhead is wrong.
Figure. Each failure with its symptom and the check that would confirm it.
What to notice. A failure without a debug step is a rumour. Pick the symptom you are seeing; the check is the next action.
Text version
Failure modes: Counting with the wrong tokeniser: The estimate is close enough to pass review and wrong at the limit. Debug by logging the provider's reported usage alongside your local count; a persistent gap means the encoding or the message overhead is wrong. | Truncating serialised output: A response is cut at the output cap, leaving invalid JSON. Debug by checking the finish reason, not the text. If it is a length stop, raise the cap or shorten the prompt; do not add a repair prompt before fixing the budget. | Numeric and identifier errors: Long numbers, IDs and dates can split into pieces that obscure place-value structure. Debug by testing the exact formats you care about; if the model mangles them, extract them with code or a tool rather than relying on generation. | Cost drift without traffic growth: Average prompt size grows because a template gained examples or a retrieved document got longer. Debug by tracking tokens per request by endpoint and by prompt version, not just total spend.
In production
Count tokens at the edge before the call and reject or trim oversized inputs with a clear error. Treat the tokeniser version as part of the prompt version, because a model upgrade can change both counts and behaviour. Reserve output space explicitly, cache counts for repeated content, and alert on tokens per request and output-to-input ratio. For cost models, use the provider's reported usage as ground truth and reconcile it against your own estimates weekly.
Common misconceptions
People think: One token is roughly one word.
Actually: A token is a subword unit. Common English words are often one token, but long words, names, code and other languages split into several.
People think: A token count is portable across providers.
Actually: Each model has its own vocabulary and merge rules. Counts differ, sometimes substantially, so always count with the target model's tokeniser.
People think: If the text fits in characters, it fits in the context window.
Actually: Context limits are in tokens. Character-based estimates routinely undercount code, JSON, and non-English text.
Interview angle
Token questions test whether you can connect a model's interface to the product's constraints. Expect "why is this request expensive", "why did the JSON break", or "how would you budget a prompt". Answer with the tokeniser, the input/output split, the reserve and the failure mode. Practise with LLM fundamentals and the prompting bank.
Sources
- Sennrich et al., Neural Machine Translation of Rare Words with Subword Units — byte-pair encoding for subword vocabularies.
- Hugging Face, Summary of the tokenizers — BPE, WordPiece and unigram compared.
- OpenAI, tiktoken — the tokeniser library used for GPT-family models.
- OpenAI, API concepts: tokens — the rough four-characters-per-token English heuristic.
- OpenAI Cookbook, How to count tokens with tiktoken — counting tokens and message overhead.
Related
- Tokenisation — the model-side view of the same mechanism.
- Subword tokenisation — BPE and unigram in depth.
- Token economics — turning counts into budgets.
- Context budget — allocating the window.
- Structured outputs — why truncation breaks parsers.