Skip to content
AI Engineer Guide.
seniorReportedcurated

If your vocabulary is 100K tokens, how do you reason about whether that is the right size, and what does it cost?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How does weight tying change the economics of a large vocabulary?
  • Tying points the input embedding and the output projection at the same matrix, so the vocabulary term is paid once instead of twice.
  • The saving scales with how large the vocabulary matrix is relative to the model, so it dominates a small model's budget and barely registers on a large one.
  • It does not change the softmax cost per position, which still spans the whole vocabulary on every step.
How would you decide whether 100k is right for your traffic?
  • Measure fertility, tokens per word, on held-out text for every language, script and content type you serve, and look at the slices that spend visibly more tokens.
  • Convert that into cost per request with the input and output rates, since a vocabulary choice that shortens sequences also changes what fits in context.
  • Finish with downstream quality on your own tasks, because a benchmark average can hide a regression on the domain you actually serve.
  • Re-run the decision when the traffic mix changes, since a vocabulary tuned for English chat can be wrong for a code or multilingual product.

Red flags

  • Says only that bigger vocabularies are more accurate
  • Ignores the softmax and embedding costs entirely

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals