Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 11 of 33

Vocabulary

0 of 33 complete

Why it matters

The vocabulary is a contract between text and parameters. It fixes which strings the model can represent in one step, how much of the output distribution is spent on rare words, and how large the final projection is. Because the embedding matrix and the output head are tied to it, changing the vocabulary is a weight surgery rather than a configuration change, and a mismatch between the vocabulary a model was trained with and the one it is served with produces fluent nonsense.

Mental model

The vocabulary is a keyboard with a fixed number of keys: more keys let you type faster, but each key gets less practice and the keyboard gets bigger. The metaphor breaks because keys can overlap — a word may be typed with several keys or with one — so adding a key does not simply add a symbol.

How it works

The embedding table · one row per idInteractive

hover or focus a cell

−1.00+1.00midpoint 0
id (row index) ↓dim →
d₁d₂d₃
pad
bos
eos
unk
the
mat

Hover or focus a cell to read its exact value. Shade and printed number encode the same quantity.

ids shown
6
dimensions
3
table
6 × 3 = 18
row index
the id itself

Step 1 of 3

One row per id

The embedding layer is a lookup, not a computation: id 3 selects the fourth row and nothing else. The table's height is the vocabulary size and its width is the hidden dimension, so either choice is visible in the parameter count.

Next: Special ids are rows too

Step 1/3 · One row per id

Figure. Six ids and three embedding dimensions, illustrative vectors. Each row is one vocabulary entry and each column one hidden dimension, so the table's shape is V × d: here 6 × 3 = 18 weights. The id is the row index, which is why a token's vector moves as one block when the id changes and why adding a token adds a row rather than editing an existing one. The values are invented; the shape and the indexing are the claim.

What to notice. With tied embeddings the same 18 numbers also score the output: the logit for id i is the dot product of row i with the final hidden state, so the input matrix and the output head are one matrix rather than two. Pad is held at zero in this example, which is a choice, not a requirement; bos, eos and unk are learned rows like any other. Six ids stand in for tens of thousands, and the figure cannot show how often each row was updated, which is what separates a well-trained row from a rare one. The printed vectors are not from a model and carry no semantic distance.

Text version

A six by three embedding table. Rows are pad, bos, eos, unk, the and mat; columns are dimensions d1, d2 and d3. Pad prints 0.00, 0.00 and 0.00; bos prints 0.40, −0.70 and 0.20; eos prints −0.30, 0.50 and 0.80; unk prints 0.10, 0.20 and −0.60; the prints 0.90, −0.20 and 0.40; mat prints −0.50, 0.70 and −0.80. Shade runs teal below zero to accent above on one scale from −1 to +1. Steps highlight the four special rows and then id 4, the, as the tied input-and-output example; the readout reports the id, the dimension and the printed weight.

  1. Choose the size from the corpus and the language mix, remembering that embedding parameters scale as vocabulary times hidden width.
  2. Decide the special-token set and their ids before training, since checkpoints and downstream code assume fixed positions.
  3. Reserve ids for padding, sequence start, sequence end and unknown input, and make the loss ignore padding targets.
  4. Check coverage by measuring how often the unknown token appears on a held-out sample of real text.
  5. Inspect the token frequency distribution: ids seen only a handful of times will have weak embeddings and unreliable output rows.
  6. If you extend the vocabulary later, resize the embedding and head, initialise new rows from the mean of similar tokens, and fine-tune.

Trade-offs

ChoiceGood forCostFails when
Large vocabularyShorter sequences, cheaper attention per wordBigger embedding and head, sparse gradient per idCorpus is small or long-tailed
Small vocabularyDense training signal for every idLonger sequences and more steps per documentContext window is the constraint
Tied embedding and output headFewer parameters, better rare-token sharingLess freedom to specialise input and output spacesInput and output distributions differ sharply
Reserved special tokensExplicit control over padding and boundariesIds that carry no semantics and consume budgetThe training code never masks them out of the loss
Trade-off mapInteractive

Good for

Shorter sequences, cheaper attention per word

Cost

Bigger embedding and head, sparse gradient per id

Fails when

Corpus is small or long-tailed

  1. Corpus is small or long-tailed
  2. Context window is the constraint
  3. Input and output distributions differ sharply
  4. The training code never masks them out of the loss

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Large vocabulary is good for Shorter sequences, cheaper attention per word, costs Bigger embedding and head, sparse gradient per id, and fails when Corpus is small or long-tailed. Small vocabulary is good for Dense training signal for every id, costs Longer sequences and more steps per document, and fails when Context window is the constraint. Tied embedding and output head is good for Fewer parameters, better rare-token sharing, costs Less freedom to specialise input and output spaces, and fails when Input and output distributions differ sharply. Reserved special tokens is good for Explicit control over padding and boundaries, costs Ids that carry no semantics and consume budget, and fails when The training code never masks them out of the loss

Failure modes

  • Unknown-token flooding. A large share of production text maps to the unknown id and the model loses information before it starts. Debug: measure unknown-token rate per input source on a sample of live traffic.
  • Off-by-one special ids. The beginning-of-sequence id differs between tokenizer config and training code, so prompts are framed wrongly. Debug: decode the exact id sequence sent to the model and read it back as text.
  • Untrained new rows. After a vocabulary extension the new embeddings are random and outputs degrade for those tokens. Debug: compare loss on examples containing new tokens before and after the fine-tuning stage.

In production

Monitor unknown-token rate and average tokens per request by source; both move when input distribution shifts. Pin the vocabulary file with the checkpoint and refuse to load weights whose embedding rows do not match the tokenizer size.

Common misconceptions

People think: A bigger vocabulary is strictly more efficient.

Actually: It shortens sequences but enlarges the embedding and output matrices and starves rare ids of gradient, so the right size depends on corpus size and language coverage.

People think: Special tokens are just conventions.

Actually: Their ids are indexed by position in the embedding matrix and referenced by training and inference code; changing them invalidates a checkpoint.

Interview angle

A strong answer treats the vocabulary as an interface with fixed dimensions, then discusses the size trade-off in terms of sequence length, parameter count and gradient density. It should mention special tokens and what a resize requires. Weak answers equate vocabulary size with model quality and cannot say which two matrices change when a token is added.

Sources