seniorReportedcurated
If your vocabulary is 100K tokens, how do you reason about whether that is the right size, and what does it cost?
30-second answer
- Vocabulary size trades sequence length against parameter and softmax cost: a bigger vocabulary packs text into fewer tokens but widens both the embedding matrix and the output distribution.
- At d_model 4096 a 100k vocabulary is about 410M embedding parameters before tying, and the final softmax touches every token id at every position.
- A token's embedding row is only updated on steps where that token appears, so a rare token receives few gradient updates and its representation stays weak; that can hurt low-resource languages and unusual domains, where much of the text is rare tokens.
- A smaller vocabulary keeps gradients dense but lengthens sequences and risks fragmenting text into pieces that carry little meaning.
- The honest test is fertility, tokens per word, measured per language and content type, plus downstream quality on your own traffic rather than a single benchmark.
Likely follow-ups
Open a question to rehearse the answer.
How does weight tying change the economics of a large vocabulary?
- Tying points the input embedding and the output projection at the same matrix, so the vocabulary term is paid once instead of twice.
- The saving scales with how large the vocabulary matrix is relative to the model, so it dominates a small model's budget and barely registers on a large one.
- It does not change the softmax cost per position, which still spans the whole vocabulary on every step.
How would you decide whether 100k is right for your traffic?
- Measure fertility, tokens per word, on held-out text for every language, script and content type you serve, and look at the slices that spend visibly more tokens.
- Convert that into cost per request with the input and output rates, since a vocabulary choice that shortens sequences also changes what fits in context.
- Finish with downstream quality on your own tasks, because a benchmark average can hide a regression on the domain you actually serve.
- Re-run the decision when the traffic mix changes, since a vocabulary tuned for English chat can be wrong for a code or multilingual product.
Red flags
- Says only that bigger vocabularies are more accurate
- Ignores the softmax and embedding costs entirely
Concepts this tests
Tags
Source: github.com