midReportedcurated
Walk me through BPE training and encoding. Why start from bytes, and what does vocabulary size trade off?
30-second answer
- Byte-level BPE starts with 256 byte symbols, counts adjacent pairs weighted by word frequency, merges the most frequent pair into a new symbol, records that merge, and repeats until the vocabulary target is reached.
- The merge list is ordered, so encoding splits text into bytes and applies the recorded merges in learned order wherever they match, which makes encoding deterministic.
- Pre-tokenisation regexes stop merges crossing word boundaries, which is what keeps a common word from gluing onto the word after it.
- Larger vocabularies shorten sequences but grow the embedding and softmax; smaller ones lengthen sequences and spend context on fragments.
- Starting from bytes guarantees no out-of-vocabulary input at all, and the price is that multi-byte characters such as CJK text or emoji fragment into several tokens.
Likely follow-ups
Open a question to rehearse the answer.
Why can encoding not simply apply the most frequent merges first at inference time?
- Merge ranks are the training artefact: each rule was learned against the segmentation that earlier merges had already produced, and the rank order is what reproduces that segmentation.
- A later rule's left and right parts may themselves be symbols created earlier, so applying rules out of order can leave a rule inapplicable or applicable where it never was.
- Re-sorting merges by global frequency therefore cuts the string differently, so the same text becomes a token sequence the weights never saw during training.
- Encoding therefore replays the learned order over the byte sequence, which is also why the tokenizer has to be versioned and stored with the model.
How would you check a tokenizer is costing you context on your traffic?
- Compute tokens per word, fertility, on a live sample split by language, script and content type, and compare each slice against the training mix.
- Inspect the segmentation of the strings that matter, such as identifiers, URLs, dates and accented text, instead of trusting an aggregate average.
- Watch truncation rate and tokens per request, since a jump in either is usually a tokenizer or input-format change rather than a model regression.
- Keep a byte-level fallback so unknown characters encode instead of raising or collapsing to a single unknown id.
Red flags
- Says BPE merges characters by frequency at encode time
- Believes a larger vocabulary is always better
Concepts this tests
Tags
Source: github.com