Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 6 of 28

Tokens

0 of 28 complete

Why it matters

Tokens are the unit in which every LLM constraint is expressed. The context window is a token budget. Pricing is per input and output token. Throughput is tokens per second. Truncation happens at a token boundary, which is why a cut prompt can end mid-word or mid-JSON.

Misjudging tokens produces a specific class of bug: a prompt that passes in testing and overflows in production once a user pastes more text, a JSON response that is valid in staging and truncated in the field, or a monthly bill that is double the estimate because the content is not English prose. Counting tokens with the wrong tokeniser is a silent error: the count looks plausible and is simply not the number the provider will charge.

Mental model

A compression dictionary. The tokeniser learns frequent character sequences and assigns each one an ID; common words become one entry, rarer words split into pieces.

The analogy breaks because the dictionary is not optimised for meaning or for human readability. It is optimised for compression on a training corpus. That is why " the" with a leading space can be a single token, why a date can split into several, and why a word that is common in one language may cost multiple tokens in another.

How it works

  1. 1Normalise the text and split it into candidate pieces at whitespace and punctuation boundaries.
  2. 2Apply the vocabulary's merge rules to combine frequent adjacent pieces into larger units.
  3. 3Map each resulting piece to an integer ID from the fixed vocabulary.
  4. 4Insert special tokens that mark roles, boundaries, or tool calls.
  5. 5Send the ID sequence to the model; detokenise the output IDs back to text.

Byte-pair encoding is the common recipe. Start with a base vocabulary of characters or bytes, count adjacent pairs in a corpus, and repeatedly merge the most frequent pair into a new symbol. Each merge is one vocabulary entry. After enough merges, frequent words become single tokens and rare words remain compositions of common pieces. SentencePiece-style unigram tokenisers take a different route — they fit a probabilistic model over subword candidates and prune the vocabulary — but the interface is the same: text in, IDs out.

The vocabulary is fixed when the model is trained. A model cannot read a token it has never seen, so genuinely novel strings are decomposed into bytes or characters. This is why tokenisers handle arbitrary Unicode without an unknown-token failure, and why unusual scripts and emoji can be expensive: each may be several tokens.

Detokenisation is not always the exact inverse of tokenisation. Whitespace handling differs by tokeniser, and a token may encode a leading space. Round-tripping text through tokens and back is usually safe for content, but do not rely on it for byte-exact reproduction.

Quick facts

Unit
Subword ID
Vocabulary
Fixed at training time
Characters per token
Varies by tokenizer and language
Code and JSON
Often more tokens per character
Pricing unit
Input and output tokens
Truncation
Always at a token boundary

OpenAI documents a rough English heuristic of about four characters per token. Treat it as a planning default only, never as a billing estimate.

The arithmetic that matters

For a request with TinT_{\text{in}} input tokens and ToutT_{\text{out}} output tokens:

cost=Tin⋅pin+Tout⋅pout\text{cost} = T_{\text{in}}\cdot p_{\text{in}} + T_{\text{out}}\cdot p_{\text{out}}

and the space left for output is:

Tout,max=W−Tin−TmarginT_{\text{out,max}} = W - T_{\text{in}} - T_{\text{margin}}

where WW is the context window and TmarginT_{\text{margin}} is headroom for tokeniser drift, tool schemas, and provider-side framing tokens you did not count. A margin of zero is a truncation bug waiting for a longer input. Output tokens usually cost more than input tokens, so a cheaper prompt does not always mean a cheaper request: a verbose answer can dominate the bill.

Counting before you send

import tiktoken
 
def count_tokens(text: str, model_name: str) -> int:
    # The encoding is tied to the model family; never reuse a count across models.
    enc = tiktoken.encoding_for_model(model_name)
    return len(enc.encode(text))
 
def fits(text: str, model_name: str, window: int, reserve: int = 512) -> bool:
    return count_tokens(text, model_name) + reserve <= window

The exact function names differ across tokeniser libraries, but the contract is the same: obtain the encoding for the target model, encode, and count. For a message list, count the serialised messages plus the per-message overhead the provider adds; a rough count of the visible text understates the request.

Trade-offs

ChoiceGood forCostFails when
Byte-level BPEArbitrary Unicode, code, no unknown tokensLonger sequences for non-English textYou need short sequences in many scripts
SentencePiece unigramMultilingual coverage, language-agnostic pipelinesModel-specific behaviour, less transparentYou assume counts transfer between models
Word-level vocabularyShort sequences for common wordsHuge vocabulary, unknown-token failuresInput is code, names, or misspellings
Character-levelTiny vocabulary, robust to noveltyVery long sequences, slow generationContext or latency is constrained
Trade-off mapInteractive

Good for

Arbitrary Unicode, code, no unknown tokens

Cost

Longer sequences for non-English text

Fails when

You need short sequences in many scripts

  1. You need short sequences in many scripts
  2. You assume counts transfer between models
  3. Input is code, names, or misspellings
  4. Context or latency is constrained

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Byte-level BPE is good for Arbitrary Unicode, code, no unknown tokens, costs Longer sequences for non-English text, and fails when You need short sequences in many scripts. SentencePiece unigram is good for Multilingual coverage, language-agnostic pipelines, costs Model-specific behaviour, less transparent, and fails when You assume counts transfer between models. Word-level vocabulary is good for Short sequences for common words, costs Huge vocabulary, unknown-token failures, and fails when Input is code, names, or misspellings. Character-level is good for Tiny vocabulary, robust to novelty, costs Very long sequences, slow generation, and fails when Context or latency is constrained

Failure modes

Counting with the wrong tokeniser. The estimate is close enough to pass review and wrong at the limit. Debug by logging the provider's reported usage alongside your local count; a persistent gap means the encoding or the message overhead is wrong.

Truncating serialised output. A response is cut at the output cap, leaving invalid JSON. Debug by checking the finish reason, not the text. If it is a length stop, raise the cap or shorten the prompt; do not add a repair prompt before fixing the budget.

Numeric and identifier errors. Long numbers, IDs and dates can split into pieces that obscure place-value structure. Debug by testing the exact formats you care about; if the model mangles them, extract them with code or a tool rather than relying on generation.

Cost drift without traffic growth. Average prompt size grows because a template gained examples or a retrieved document got longer. Debug by tracking tokens per request by endpoint and by prompt version, not just total spend.

Failure modes, one at a timeInteractive

Symptom

The estimate is close enough to pass review and wrong at the limit.

Debug step

Logging the provider's reported usage alongside your local count; a persistent gap means the encoding or the message overhead is wrong.

Figure. Each failure with its symptom and the check that would confirm it.

What to notice. A failure without a debug step is a rumour. Pick the symptom you are seeing; the check is the next action.

Text version

Failure modes: Counting with the wrong tokeniser: The estimate is close enough to pass review and wrong at the limit. Debug by logging the provider's reported usage alongside your local count; a persistent gap means the encoding or the message overhead is wrong. | Truncating serialised output: A response is cut at the output cap, leaving invalid JSON. Debug by checking the finish reason, not the text. If it is a length stop, raise the cap or shorten the prompt; do not add a repair prompt before fixing the budget. | Numeric and identifier errors: Long numbers, IDs and dates can split into pieces that obscure place-value structure. Debug by testing the exact formats you care about; if the model mangles them, extract them with code or a tool rather than relying on generation. | Cost drift without traffic growth: Average prompt size grows because a template gained examples or a retrieved document got longer. Debug by tracking tokens per request by endpoint and by prompt version, not just total spend.

In production

Count tokens at the edge before the call and reject or trim oversized inputs with a clear error. Treat the tokeniser version as part of the prompt version, because a model upgrade can change both counts and behaviour. Reserve output space explicitly, cache counts for repeated content, and alert on tokens per request and output-to-input ratio. For cost models, use the provider's reported usage as ground truth and reconcile it against your own estimates weekly.

Common misconceptions

People think: One token is roughly one word.

Actually: A token is a subword unit. Common English words are often one token, but long words, names, code and other languages split into several.

People think: A token count is portable across providers.

Actually: Each model has its own vocabulary and merge rules. Counts differ, sometimes substantially, so always count with the target model's tokeniser.

People think: If the text fits in characters, it fits in the context window.

Actually: Context limits are in tokens. Character-based estimates routinely undercount code, JSON, and non-English text.

Interview angle

Token questions test whether you can connect a model's interface to the product's constraints. Expect "why is this request expensive", "why did the JSON break", or "how would you budget a prompt". Answer with the tokeniser, the input/output split, the reserve and the failure mode. Practise with LLM fundamentals and the prompting bank.

Sources