Why it matters
Classification and language modelling need a loss that rewards assigning high probability to the correct answer and punishes confident mistakes. Cross-entropy does exactly that, and its gradient through a softmax is clean: predicted probability minus one-hot target. Entropy also gives a vocabulary for uncertainty and information, which is why the same quantity appears in label smoothing, active learning and compression arguments.
Mental model
Entropy is the average number of yes/no questions needed to identify an outcome, if you ask them optimally. The metaphor breaks because the questions are not free-form — the code length is fixed by the assumed distribution, so a wrong model costs more questions than the entropy suggests.
How it works
- 1Convert probabilities to logarithms, usually base e for nats or base 2 for bits.
- 2Compute entropy as the negative expected log probability under the distribution itself.
- 3Compute cross-entropy using the true distribution's probabilities against the model's log probabilities.
- 4Recognise that cross-entropy equals entropy plus KL divergence, so the entropy term is constant with respect to the model.
- 5Apply the loss only to valid targets, masking padding and ignoring positions with no label.
- 6Report bits per character when comparing models with different tokenizers, since per-token loss is not portable.
Step 1 of 6
Convert probabilities to logarithms, usually base e for nats or base 2 for bits.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Convert probabilities to logarithms, usually base e for nats or base 2 for bits. 2. Compute entropy as the negative expected log probability under the distribution itself. 3. Compute cross-entropy using the true distribution's probabilities against the model's log… 4. Recognise that cross-entropy equals entropy plus KL divergence, so the entropy term is co… 5. Apply the loss only to valid targets, masking padding and ignoring positions with no labe… 6. Report bits per character when comparing models with different tokenizers, since per-toke…
- Convert probabilities to logarithms, usually base e for nats or base 2 for bits.
- Compute entropy as the negative expected log probability under the distribution itself.
- Compute cross-entropy using the true distribution's probabilities against the model's log probabilities.
- Recognise that cross-entropy equals entropy plus KL divergence, so the entropy term is constant with respect to the model.
- Apply the loss only to valid targets, masking padding and ignoring positions with no label.
- Report bits per character when comparing models with different tokenizers, since per-token loss is not portable.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Cross-entropy loss | Proper scoring rule, strong gradients, standard | Unbounded penalty for confident errors | Labels are noisy or mislabelled |
| Label smoothing | Guards against overconfidence | Caps the achievable confidence and changes calibration | You need sharp probabilities |
| Mean squared error on probabilities | Bounded gradients | Weak learning signal when predictions are very wrong | The task is classification |
| Focal loss | Focuses on hard examples | Extra hyperparameters, can ignore easy classes | Classes are balanced |
Good for
Proper scoring rule, strong gradients, standard
Cost
Unbounded penalty for confident errors
Fails when
Labels are noisy or mislabelled
- Labels are noisy or mislabelled
- You need sharp probabilities
- The task is classification
- Classes are balanced
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Cross-entropy loss is good for Proper scoring rule, strong gradients, standard, costs Unbounded penalty for confident errors, and fails when Labels are noisy or mislabelled. Label smoothing is good for Guards against overconfidence, costs Caps the achievable confidence and changes calibration, and fails when You need sharp probabilities. Mean squared error on probabilities is good for Bounded gradients, costs Weak learning signal when predictions are very wrong, and fails when The task is classification. Focal loss is good for Focuses on hard examples, costs Extra hyperparameters, can ignore easy classes, and fails when Classes are balanced
Failure modes
- Log of zero. A predicted probability of exactly zero produces infinite loss. Debug: check for zero or denormal probabilities after the softmax and clip them to a small epsilon.
- Loss over padding. Padding positions contribute to the average and dilute the signal. Debug: compare the number of scored positions with the number of real tokens.
- Mixing log bases. Loss values differ by a constant factor and thresholds tuned on one scale fail on the other. Debug: confirm the base in the loss implementation and in the reported metric.
In production
Track cross-entropy on a fixed evaluation set and bits per character for cross-model comparison. Watch for a rising fraction of very confident wrong predictions, which cross-entropy punishes heavily but accuracy does not reveal.
Common misconceptions
People think: Cross-entropy and accuracy measure the same thing.
Actually: Accuracy counts correct decisions; cross-entropy scores the probability assigned to the truth, so a model can be accurate and badly calibrated at once.
People think: Lower entropy always means a better model.
Actually: Entropy of the predictions measures confidence, not correctness; a confidently wrong model has low entropy and terrible loss.
Interview angle
A strong answer defines entropy, derives cross-entropy as entropy plus KL, and explains why the gradient through softmax is simple. It should discuss masking and the bits-per-character correction. Weak answers say entropy is 'disorder' and cannot say why cross-entropy is preferred to squared error for classification.
Sources
- Shannon, A Mathematical Theory of Communication — the original definition of entropy.
- MacKay, Information Theory, Inference, and Learning Algorithms — entropy, KL divergence and coding interpretations.
- scikit-learn, log_loss — cross-entropy as an implemented classification loss.