Why it matters
A model cannot optimise accuracy directly because accuracy is piecewise constant and has no useful gradient. The loss is a differentiable surrogate that must be close enough to the real objective to be worth minimising. Choosing it is therefore a modelling decision: squared error punishes large deviations heavily, absolute error tolerates outliers, cross-entropy punishes confident mistakes, and each implies a different notion of what a good prediction is.
Mental model
A loss is a scoring rule for a bet: it decides how much you lose for being wrong and how much more for being confidently wrong. The metaphor breaks because the model never sees the payout table as a whole — it only follows the local slope, so a badly shaped loss can be minimised in a way that does not help the product.
How it works
- 1State the target type: continuous unbounded, continuous positive, binary, multi-class or ranked.
- 2Pick the likelihood that matches it, then take the negative log likelihood as the loss.
- 3Choose the reduction deliberately, since summing over a variable-length batch weights long examples more.
- 4Check class or token weighting when the data is imbalanced, and record the weights with the model.
- 5Compare the loss curve against the product metric on a validation set; a diverging pair means the surrogate is wrong.
- 6Test the loss on hand-built edge cases: perfect prediction, uniform prediction and a confidently wrong prediction.
Step 1 of 6
State the target type: continuous unbounded, continuous positive, binary, multi-class or ranked.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. State the target type: continuous unbounded, continuous positive, binary, multi-class or … 2. Pick the likelihood that matches it, then take the negative log likelihood as the loss. 3. Choose the reduction deliberately, since summing over a variable-length batch weights lon… 4. Check class or token weighting when the data is imbalanced, and record the weights with t… 5. Compare the loss curve against the product metric on a validation set; a diverging pair m… 6. Test the loss on hand-built edge cases: perfect prediction, uniform prediction and a conf…
- State the target type: continuous unbounded, continuous positive, binary, multi-class or ranked.
- Pick the likelihood that matches it, then take the negative log likelihood as the loss.
- Choose the reduction deliberately, since summing over a variable-length batch weights long examples more.
- Check class or token weighting when the data is imbalanced, and record the weights with the model.
- Compare the loss curve against the product metric on a validation set; a diverging pair means the surrogate is wrong.
- Test the loss on hand-built edge cases: perfect prediction, uniform prediction and a confidently wrong prediction.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Mean squared error | Smooth gradients, punishes large errors | Dominated by outliers, assumes Gaussian noise | Targets have heavy tails |
| Mean absolute error | Robust to outliers, interpretable units | Constant gradient near zero, slower convergence | You need precise small corrections |
| Cross-entropy | Proper scoring rule, strong classification gradients | Unbounded penalty for confident mistakes | Labels are noisy |
| Focal or weighted loss | Focus on hard or rare examples | Extra hyperparameters, risk of ignoring easy cases | Classes are balanced and clean |
Good for
Smooth gradients, punishes large errors
Cost
Dominated by outliers, assumes Gaussian noise
Fails when
Targets have heavy tails
- Targets have heavy tails
- You need precise small corrections
- Labels are noisy
- Classes are balanced and clean
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Mean squared error is good for Smooth gradients, punishes large errors, costs Dominated by outliers, assumes Gaussian noise, and fails when Targets have heavy tails. Mean absolute error is good for Robust to outliers, interpretable units, costs Constant gradient near zero, slower convergence, and fails when You need precise small corrections. Cross-entropy is good for Proper scoring rule, strong classification gradients, costs Unbounded penalty for confident mistakes, and fails when Labels are noisy. Focal or weighted loss is good for Focus on hard or rare examples, costs Extra hyperparameters, risk of ignoring easy cases, and fails when Classes are balanced and clean
Failure modes
- Loss and metric disagree. Validation loss improves while the product metric stalls or falls. Debug: plot both on the same run and inspect the examples where they diverge.
- Reduction changed silently. Switching from mean to sum multiplies the effective learning rate by the batch size. Debug: compare the loss magnitude before and after the change on an identical batch.
- Targets outside the loss domain. Probabilities or logits are passed to a loss expecting the other, producing NaN or nonsense. Debug: assert the value range of predictions and targets at the loss boundary.
In production
Log the loss with its reduction and any class weights attached to the run, because a loss without those details is not reproducible. Watch per-slice loss rather than the aggregate, since a loss that improves overall can degrade the segment you care about.
Common misconceptions
People think: A lower loss always means a better model.
Actually: The loss is a surrogate; it can improve while the product metric stalls, especially when the training distribution differs from deployment.
People think: Any differentiable loss will do if you train long enough.
Actually: The shape determines which errors dominate the gradient, so a mismatched loss converges to the wrong behaviour rather than to a slower version of the right one.
Interview angle
A strong answer derives the loss from the target distribution, discusses reduction and weighting, and separates the surrogate from the reported metric. It should give an example where they diverge. Weak answers list losses by task without saying what assumption each encodes or how class imbalance changes the effective objective.
Sources
- Goodfellow, Bengio & Courville, Deep Learning — loss functions as maximum likelihood under an assumed distribution.
- scikit-learn, Metrics and scoring — the distinction between losses and evaluation metrics.
- Lin et al., Focal Loss for Dense Object Detection — reshaping a loss to focus on hard examples.