Why it matters
The vocabulary is a contract between text and parameters. It fixes which strings the model can represent in one step, how much of the output distribution is spent on rare words, and how large the final projection is. Because the embedding matrix and the output head are tied to it, changing the vocabulary is a weight surgery rather than a configuration change, and a mismatch between the vocabulary a model was trained with and the one it is served with produces fluent nonsense.
Mental model
The vocabulary is a keyboard with a fixed number of keys: more keys let you type faster, but each key gets less practice and the keyboard gets bigger. The metaphor breaks because keys can overlap — a word may be typed with several keys or with one — so adding a key does not simply add a symbol.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value. Shade and printed number encode the same quantity.
- ids shown
- 6
- dimensions
- 3
- table
- 6 × 3 = 18
- row index
- the id itself
Step 1 of 3
One row per id
The embedding layer is a lookup, not a computation: id 3 selects the fourth row and nothing else. The table's height is the vocabulary size and its width is the hidden dimension, so either choice is visible in the parameter count.
Next: Special ids are rows too
Figure. Six ids and three embedding dimensions, illustrative vectors. Each row is one vocabulary entry and each column one hidden dimension, so the table's shape is V × d: here 6 × 3 = 18 weights. The id is the row index, which is why a token's vector moves as one block when the id changes and why adding a token adds a row rather than editing an existing one. The values are invented; the shape and the indexing are the claim.
What to notice. With tied embeddings the same 18 numbers also score the output: the logit for id i is the dot product of row i with the final hidden state, so the input matrix and the output head are one matrix rather than two. Pad is held at zero in this example, which is a choice, not a requirement; bos, eos and unk are learned rows like any other. Six ids stand in for tens of thousands, and the figure cannot show how often each row was updated, which is what separates a well-trained row from a rare one. The printed vectors are not from a model and carry no semantic distance.
Text version
A six by three embedding table. Rows are pad, bos, eos, unk, the and mat; columns are dimensions d1, d2 and d3. Pad prints 0.00, 0.00 and 0.00; bos prints 0.40, −0.70 and 0.20; eos prints −0.30, 0.50 and 0.80; unk prints 0.10, 0.20 and −0.60; the prints 0.90, −0.20 and 0.40; mat prints −0.50, 0.70 and −0.80. Shade runs teal below zero to accent above on one scale from −1 to +1. Steps highlight the four special rows and then id 4, the, as the tied input-and-output example; the readout reports the id, the dimension and the printed weight.
- Choose the size from the corpus and the language mix, remembering that embedding parameters scale as vocabulary times hidden width.
- Decide the special-token set and their ids before training, since checkpoints and downstream code assume fixed positions.
- Reserve ids for padding, sequence start, sequence end and unknown input, and make the loss ignore padding targets.
- Check coverage by measuring how often the unknown token appears on a held-out sample of real text.
- Inspect the token frequency distribution: ids seen only a handful of times will have weak embeddings and unreliable output rows.
- If you extend the vocabulary later, resize the embedding and head, initialise new rows from the mean of similar tokens, and fine-tune.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Large vocabulary | Shorter sequences, cheaper attention per word | Bigger embedding and head, sparse gradient per id | Corpus is small or long-tailed |
| Small vocabulary | Dense training signal for every id | Longer sequences and more steps per document | Context window is the constraint |
| Tied embedding and output head | Fewer parameters, better rare-token sharing | Less freedom to specialise input and output spaces | Input and output distributions differ sharply |
| Reserved special tokens | Explicit control over padding and boundaries | Ids that carry no semantics and consume budget | The training code never masks them out of the loss |
Good for
Shorter sequences, cheaper attention per word
Cost
Bigger embedding and head, sparse gradient per id
Fails when
Corpus is small or long-tailed
- Corpus is small or long-tailed
- Context window is the constraint
- Input and output distributions differ sharply
- The training code never masks them out of the loss
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Large vocabulary is good for Shorter sequences, cheaper attention per word, costs Bigger embedding and head, sparse gradient per id, and fails when Corpus is small or long-tailed. Small vocabulary is good for Dense training signal for every id, costs Longer sequences and more steps per document, and fails when Context window is the constraint. Tied embedding and output head is good for Fewer parameters, better rare-token sharing, costs Less freedom to specialise input and output spaces, and fails when Input and output distributions differ sharply. Reserved special tokens is good for Explicit control over padding and boundaries, costs Ids that carry no semantics and consume budget, and fails when The training code never masks them out of the loss
Failure modes
- Unknown-token flooding. A large share of production text maps to the unknown id and the model loses information before it starts. Debug: measure unknown-token rate per input source on a sample of live traffic.
- Off-by-one special ids. The beginning-of-sequence id differs between tokenizer config and training code, so prompts are framed wrongly. Debug: decode the exact id sequence sent to the model and read it back as text.
- Untrained new rows. After a vocabulary extension the new embeddings are random and outputs degrade for those tokens. Debug: compare loss on examples containing new tokens before and after the fine-tuning stage.
In production
Monitor unknown-token rate and average tokens per request by source; both move when input distribution shifts. Pin the vocabulary file with the checkpoint and refuse to load weights whose embedding rows do not match the tokenizer size.
Common misconceptions
People think: A bigger vocabulary is strictly more efficient.
Actually: It shortens sequences but enlarges the embedding and output matrices and starves rare ids of gradient, so the right size depends on corpus size and language coverage.
People think: Special tokens are just conventions.
Actually: Their ids are indexed by position in the embedding matrix and referenced by training and inference code; changing them invalidates a checkpoint.
Interview angle
A strong answer treats the vocabulary as an interface with fixed dimensions, then discusses the size trade-off in terms of sequence length, parameter count and gradient density. It should mention special tokens and what a resize requires. Weak answers equate vocabulary size with model quality and cannot say which two matrices change when a token is added.
Sources
- Hugging Face, Summary of the tokenizers — vocabulary construction and special-token handling.
- Sennrich, Haddow & Birch, Neural Machine Translation of Rare Words with Subword Units — why a fixed subword inventory replaces open vocabularies.
- Press & Wolf, Using the Output Embedding to Improve Language Models — tying input and output embeddings and its parameter effect.