Skip to content
AI Engineer Guide.
midReportedcurated

Walk me through BPE training and encoding. Why start from bytes, and what does vocabulary size trade off?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Why can encoding not simply apply the most frequent merges first at inference time?
  • Merge ranks are the training artefact: each rule was learned against the segmentation that earlier merges had already produced, and the rank order is what reproduces that segmentation.
  • A later rule's left and right parts may themselves be symbols created earlier, so applying rules out of order can leave a rule inapplicable or applicable where it never was.
  • Re-sorting merges by global frequency therefore cuts the string differently, so the same text becomes a token sequence the weights never saw during training.
  • Encoding therefore replays the learned order over the byte sequence, which is also why the tokenizer has to be versioned and stored with the model.
How would you check a tokenizer is costing you context on your traffic?
  • Compute tokens per word, fertility, on a live sample split by language, script and content type, and compare each slice against the training mix.
  • Inspect the segmentation of the strings that matter, such as identifiers, URLs, dates and accented text, instead of trusting an aggregate average.
  • Watch truncation rate and tokens per request, since a jump in either is usually a tokenizer or input-format change rather than a model regression.
  • Keep a byte-level fallback so unknown characters encode instead of raising or collapsing to a single unknown id.

Red flags

  • Says BPE merges characters by frequency at encode time
  • Believes a larger vocabulary is always better

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals