Skip to content
AI Engineer Guide.
seniorReportedcurated

What does perplexity measure, and why can't you compare perplexity across models with different tokenizers?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How do you convert a loss of 2.0 nats per token into perplexity?
  • Perplexity is e raised to the mean loss, so 2.0 nats per token gives e² ≈ 7.4 equally likely choices per step.
  • In bits that is 2.0 / ln 2 ≈ 2.9 bits per token; divide by bytes per token to get bits per byte for cross-tokenizer comparison.
  • The conversion is exact arithmetic, but the number is only meaningful against the same tokenizer and the same evaluation text.
Why might a better model show higher perplexity on your eval set?
  • Post-training shifts the distribution toward answers people prefer, and that usually costs likelihood on raw web text, so a better assistant can score worse.
  • A different tokenizer changes the denominator: the same text becomes fewer, larger tokens, each carrying more information.
  • Contamination works the other way and makes perplexity look better than the model deserves, which is why a suspiciously low score is a warning.
  • Domain shift also matters, since a model tuned on chat can be worse than a base model on legal or code text.

Red flags

  • Compares raw perplexity across tokenizers
  • Claims lower perplexity always means a better assistant

Concepts this tests

Tags

Source: github.com

More in Maths & CS fundamentals