seniorReportedcurated
What does perplexity measure, and why can't you compare perplexity across models with different tokenizers?
30-second answer
- Perplexity is the exponential of mean negative log-likelihood per token, so it reads as the effective number of equally likely choices the model weighs at each step.
- It is tokenizer-dependent: a 200k-vocabulary model packs the same document into fewer, larger tokens, which raises per-token perplexity even when it models the text better.
- The portable comparison is bits per byte: divide nats per token by ln 2, then by bytes per token, so the unit stops depending on the vocabulary.
- Perplexity is dominated by easy tokens, collapses under contamination, and often worsens after post-training by design, so it cannot rank assistants.
- It also says nothing about instruction following or factuality, which is why a cheap intrinsic metric cannot replace task evaluation.
Likely follow-ups
Open a question to rehearse the answer.
How do you convert a loss of 2.0 nats per token into perplexity?
- Perplexity is e raised to the mean loss, so 2.0 nats per token gives e² ≈ 7.4 equally likely choices per step.
- In bits that is 2.0 / ln 2 ≈ 2.9 bits per token; divide by bytes per token to get bits per byte for cross-tokenizer comparison.
- The conversion is exact arithmetic, but the number is only meaningful against the same tokenizer and the same evaluation text.
Why might a better model show higher perplexity on your eval set?
- Post-training shifts the distribution toward answers people prefer, and that usually costs likelihood on raw web text, so a better assistant can score worse.
- A different tokenizer changes the denominator: the same text becomes fewer, larger tokens, each carrying more information.
- Contamination works the other way and makes perplexity look better than the model deserves, which is why a suspiciously low score is a warning.
- Domain shift also matters, since a model tuned on chat can be worse than a base model on legal or code text.
Red flags
- Compares raw perplexity across tokenizers
- Claims lower perplexity always means a better assistant
Concepts this tests
Tags
Source: github.com