Why it matters
A single number cannot describe a classifier, because the errors are not symmetric: a missed fraud is not the same as a false alarm, and a missed diagnosis is not the same as an unnecessary test. Metrics encode which error you care about and at what threshold. The engineering problem is that teams report the metric that is easiest to compute, then discover that it does not track the cost the business actually pays.
Mental model
Metrics are different lenses on the same confusion matrix: one magnifies false alarms, another magnifies misses. The metaphor breaks because the lenses are not neutral — choosing one changes which model wins the comparison, so the metric is part of the decision rather than a description of it.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.
- precision
- 42/60 = 0.700
- recall
- 42/50 = 0.840
- F1
- 0.764
- accuracy
- 174/200 = 0.870
Step 1 of 4
Choose the operating point
At this threshold the model predicts positive 60 times: 42 are right and 18 are false alarms. Of the 50 real positives it finds 42 and misses 8.
Next: Precision reads a column
Figure. A 2×2 confusion matrix at the deployed threshold, illustrative counts. Rows are the true class and columns the prediction; TP, FP, FN and TN label the cells. Shade saturates at 60 so the three smaller cells stay readable beside the 132 true negatives; the printed counts are exact.
What to notice. Precision divides the predicted-positive column: 42 / (42 + 18) = 0.700. Recall divides the actual-positive row: 42 / (42 + 8) = 0.840. Accuracy is 174/200 = 0.870, but it is carried by the 132 true negatives: a model that predicted negative everywhere would score 150/200 = 0.750 while catching nothing.
Text version
A two by two confusion matrix. Rows are actual positive and actual negative; columns are predicted positive and predicted negative. Actual positive gives 42 true positives and 8 false negatives; actual negative gives 18 false positives and 132 true negatives. Row totals are 50 and 150; column totals are 60 and 140. Steps highlight the precision column and the recall row, and the readout reports the cell tag, count and row support. The statistics strip prints precision 0.700, recall 0.840, F1 0.764 and accuracy 0.870.
- Build the confusion matrix at the threshold you will actually deploy, not at 0.5 by default.
- State the cost of a false positive and a false negative, then choose precision, recall or a weighted combination.
- For ranking quality independent of threshold, report PR-AUC when positives are rare and ROC-AUC when classes are balanced.
- Report a confidence interval or per-fold spread, since a single split gives a point estimate with unknown variance.
- Break metrics down by segment, because an aggregate can hide a group where the model is much worse.
- Check calibration separately with a reliability plot, since ranking well and being well calibrated are different properties.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Accuracy | Simple, interpretable when balanced | Useless under imbalance | One class dominates |
| Precision and recall | Directly tied to error costs | Needs a threshold and two numbers | You need a single ranking score |
| ROC-AUC | Threshold-free, comparable across models | Optimistic when positives are rare | The positive class is a small minority |
| PR-AUC | Sensitive to rare positives | Depends on prevalence, hard to compare across populations | Prevalence changes between datasets |
Good for
Simple, interpretable when balanced
Cost
Useless under imbalance
Fails when
One class dominates
- One class dominates
- You need a single ranking score
- The positive class is a small minority
- Prevalence changes between datasets
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Accuracy is good for Simple, interpretable when balanced, costs Useless under imbalance, and fails when One class dominates. Precision and recall is good for Directly tied to error costs, costs Needs a threshold and two numbers, and fails when You need a single ranking score. ROC-AUC is good for Threshold-free, comparable across models, costs Optimistic when positives are rare, and fails when The positive class is a small minority. PR-AUC is good for Sensitive to rare positives, costs Depends on prevalence, hard to compare across populations, and fails when Prevalence changes between datasets
Failure modes
- Accuracy on imbalanced data. A model that predicts the majority class looks strong and catches nothing. Debug: compare accuracy with the majority-class baseline and inspect recall on the minority class.
- Threshold tuned on the wrong prevalence. Precision collapses after deployment because positives are rarer than in the validation set. Debug: recompute the confusion matrix at the deployment prevalence.
- AUC compared across populations. Two models look different because their evaluation sets have different class balance. Debug: fix the evaluation set or report precision at a fixed recall.
In production
Monitor precision and recall at the deployed threshold, not AUC, because the threshold is what users experience. Track the predicted positive rate against the observed rate and re-tune when prevalence shifts.
Common misconceptions
People think: A higher AUC means a better model for your use case.
Actually: AUC measures ranking across all thresholds; the deployed operating point and the cost of errors decide usefulness.
People think: F1 is the safe default for imbalance.
Actually: F1 weights precision and recall equally, which is a cost assumption; if false negatives are far worse, F1 is the wrong objective.
Interview angle
A strong answer starts from error costs, picks a metric and threshold to match, and separates ranking from calibration. It should name the imbalance trap in ROC-AUC. Weak answers recite the definitions of precision and recall without saying which one to optimise or how prevalence changes the interpretation.
Sources
- scikit-learn, Metrics and scoring — definitions and averaging strategies for classification metrics.
- Saito & Rehmsmeier, The Precision-Recall Plot Is More Informative than the ROC Plot — why PR curves suit imbalanced data.
- Davis & Goadrich, The Relationship Between Precision-Recall and ROC Curves — the formal correspondence and its limits.