Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 10 of 27

Entropy

0 of 27 complete

Why it matters

Classification and language modelling need a loss that rewards assigning high probability to the correct answer and punishes confident mistakes. Cross-entropy does exactly that, and its gradient through a softmax is clean: predicted probability minus one-hot target. Entropy also gives a vocabulary for uncertainty and information, which is why the same quantity appears in label smoothing, active learning and compression arguments.

Mental model

Entropy is the average number of yes/no questions needed to identify an outcome, if you ask them optimally. The metaphor breaks because the questions are not free-form — the code length is fixed by the assumed distribution, so a wrong model costs more questions than the entropy suggests.

How it works

Mechanism, step by stepInteractive
  1. 1Convert probabilities to logarithms, usually base e for nats or base 2 for bits.
  2. 2Compute entropy as the negative expected log probability under the distribution itself.
  3. 3Compute cross-entropy using the true distribution's probabilities against the model's log probabilities.
  4. 4Recognise that cross-entropy equals entropy plus KL divergence, so the entropy term is constant with respect to the model.
  5. 5Apply the loss only to valid targets, masking padding and ignoring positions with no label.
  6. 6Report bits per character when comparing models with different tokenizers, since per-token loss is not portable.

Step 1 of 6

Convert probabilities to logarithms, usually base e for nats or base 2 for bits.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Convert probabilities to logarithms, usually base e for nats or base 2 for bits. 2. Compute entropy as the negative expected log probability under the distribution itself. 3. Compute cross-entropy using the true distribution's probabilities against the model's log… 4. Recognise that cross-entropy equals entropy plus KL divergence, so the entropy term is co… 5. Apply the loss only to valid targets, masking padding and ignoring positions with no labe… 6. Report bits per character when comparing models with different tokenizers, since per-toke…

  1. Convert probabilities to logarithms, usually base e for nats or base 2 for bits.
  2. Compute entropy as the negative expected log probability under the distribution itself.
  3. Compute cross-entropy using the true distribution's probabilities against the model's log probabilities.
  4. Recognise that cross-entropy equals entropy plus KL divergence, so the entropy term is constant with respect to the model.
  5. Apply the loss only to valid targets, masking padding and ignoring positions with no label.
  6. Report bits per character when comparing models with different tokenizers, since per-token loss is not portable.

Trade-offs

ChoiceGood forCostFails when
Cross-entropy lossProper scoring rule, strong gradients, standardUnbounded penalty for confident errorsLabels are noisy or mislabelled
Label smoothingGuards against overconfidenceCaps the achievable confidence and changes calibrationYou need sharp probabilities
Mean squared error on probabilitiesBounded gradientsWeak learning signal when predictions are very wrongThe task is classification
Focal lossFocuses on hard examplesExtra hyperparameters, can ignore easy classesClasses are balanced
Trade-off mapInteractive

Good for

Proper scoring rule, strong gradients, standard

Cost

Unbounded penalty for confident errors

Fails when

Labels are noisy or mislabelled

  1. Labels are noisy or mislabelled
  2. You need sharp probabilities
  3. The task is classification
  4. Classes are balanced

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Cross-entropy loss is good for Proper scoring rule, strong gradients, standard, costs Unbounded penalty for confident errors, and fails when Labels are noisy or mislabelled. Label smoothing is good for Guards against overconfidence, costs Caps the achievable confidence and changes calibration, and fails when You need sharp probabilities. Mean squared error on probabilities is good for Bounded gradients, costs Weak learning signal when predictions are very wrong, and fails when The task is classification. Focal loss is good for Focuses on hard examples, costs Extra hyperparameters, can ignore easy classes, and fails when Classes are balanced

Failure modes

  • Log of zero. A predicted probability of exactly zero produces infinite loss. Debug: check for zero or denormal probabilities after the softmax and clip them to a small epsilon.
  • Loss over padding. Padding positions contribute to the average and dilute the signal. Debug: compare the number of scored positions with the number of real tokens.
  • Mixing log bases. Loss values differ by a constant factor and thresholds tuned on one scale fail on the other. Debug: confirm the base in the loss implementation and in the reported metric.

In production

Track cross-entropy on a fixed evaluation set and bits per character for cross-model comparison. Watch for a rising fraction of very confident wrong predictions, which cross-entropy punishes heavily but accuracy does not reveal.

Common misconceptions

People think: Cross-entropy and accuracy measure the same thing.

Actually: Accuracy counts correct decisions; cross-entropy scores the probability assigned to the truth, so a model can be accurate and badly calibrated at once.

People think: Lower entropy always means a better model.

Actually: Entropy of the predictions measures confidence, not correctness; a confidently wrong model has low entropy and terrible loss.

Interview angle

A strong answer defines entropy, derives cross-entropy as entropy plus KL, and explains why the gradient through softmax is simple. It should discuss masking and the bits-per-character correction. Weak answers say entropy is 'disorder' and cannot say why cross-entropy is preferred to squared error for classification.

Sources