Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 23 of 27

Classification metrics

0 of 27 complete

Why it matters

A single number cannot describe a classifier, because the errors are not symmetric: a missed fraud is not the same as a false alarm, and a missed diagnosis is not the same as an unnecessary test. Metrics encode which error you care about and at what threshold. The engineering problem is that teams report the metric that is easiest to compute, then discover that it does not track the cost the business actually pays.

Mental model

Metrics are different lenses on the same confusion matrix: one magnifies false alarms, another magnifies misses. The metaphor breaks because the lenses are not neutral — choosing one changes which model wins the comparison, so the metric is part of the decision rather than a description of it.

How it works

Confusion matrix · precision and recallInteractive

hover or focus a cell

060+
true class ↓prediction →
predicted positivepredicted negativesupport
actual positive50
actual negative150
total60140

Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.

precision
42/60 = 0.700
recall
42/50 = 0.840
F1
0.764
accuracy
174/200 = 0.870

Step 1 of 4

Choose the operating point

At this threshold the model predicts positive 60 times: 42 are right and 18 are false alarms. Of the 50 real positives it finds 42 and misses 8.

Next: Precision reads a column

Step 1/4 · Choose the operating point

Figure. A 2×2 confusion matrix at the deployed threshold, illustrative counts. Rows are the true class and columns the prediction; TP, FP, FN and TN label the cells. Shade saturates at 60 so the three smaller cells stay readable beside the 132 true negatives; the printed counts are exact.

What to notice. Precision divides the predicted-positive column: 42 / (42 + 18) = 0.700. Recall divides the actual-positive row: 42 / (42 + 8) = 0.840. Accuracy is 174/200 = 0.870, but it is carried by the 132 true negatives: a model that predicted negative everywhere would score 150/200 = 0.750 while catching nothing.

Text version

A two by two confusion matrix. Rows are actual positive and actual negative; columns are predicted positive and predicted negative. Actual positive gives 42 true positives and 8 false negatives; actual negative gives 18 false positives and 132 true negatives. Row totals are 50 and 150; column totals are 60 and 140. Steps highlight the precision column and the recall row, and the readout reports the cell tag, count and row support. The statistics strip prints precision 0.700, recall 0.840, F1 0.764 and accuracy 0.870.

  1. Build the confusion matrix at the threshold you will actually deploy, not at 0.5 by default.
  2. State the cost of a false positive and a false negative, then choose precision, recall or a weighted combination.
  3. For ranking quality independent of threshold, report PR-AUC when positives are rare and ROC-AUC when classes are balanced.
  4. Report a confidence interval or per-fold spread, since a single split gives a point estimate with unknown variance.
  5. Break metrics down by segment, because an aggregate can hide a group where the model is much worse.
  6. Check calibration separately with a reliability plot, since ranking well and being well calibrated are different properties.

Trade-offs

ChoiceGood forCostFails when
AccuracySimple, interpretable when balancedUseless under imbalanceOne class dominates
Precision and recallDirectly tied to error costsNeeds a threshold and two numbersYou need a single ranking score
ROC-AUCThreshold-free, comparable across modelsOptimistic when positives are rareThe positive class is a small minority
PR-AUCSensitive to rare positivesDepends on prevalence, hard to compare across populationsPrevalence changes between datasets
Trade-off mapInteractive

Good for

Simple, interpretable when balanced

Cost

Useless under imbalance

Fails when

One class dominates

  1. One class dominates
  2. You need a single ranking score
  3. The positive class is a small minority
  4. Prevalence changes between datasets

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Accuracy is good for Simple, interpretable when balanced, costs Useless under imbalance, and fails when One class dominates. Precision and recall is good for Directly tied to error costs, costs Needs a threshold and two numbers, and fails when You need a single ranking score. ROC-AUC is good for Threshold-free, comparable across models, costs Optimistic when positives are rare, and fails when The positive class is a small minority. PR-AUC is good for Sensitive to rare positives, costs Depends on prevalence, hard to compare across populations, and fails when Prevalence changes between datasets

Failure modes

  • Accuracy on imbalanced data. A model that predicts the majority class looks strong and catches nothing. Debug: compare accuracy with the majority-class baseline and inspect recall on the minority class.
  • Threshold tuned on the wrong prevalence. Precision collapses after deployment because positives are rarer than in the validation set. Debug: recompute the confusion matrix at the deployment prevalence.
  • AUC compared across populations. Two models look different because their evaluation sets have different class balance. Debug: fix the evaluation set or report precision at a fixed recall.

In production

Monitor precision and recall at the deployed threshold, not AUC, because the threshold is what users experience. Track the predicted positive rate against the observed rate and re-tune when prevalence shifts.

Common misconceptions

People think: A higher AUC means a better model for your use case.

Actually: AUC measures ranking across all thresholds; the deployed operating point and the cost of errors decide usefulness.

People think: F1 is the safe default for imbalance.

Actually: F1 weights precision and recall equally, which is a cost assumption; if false negatives are far worse, F1 is the wrong objective.

Interview angle

A strong answer starts from error costs, picks a metric and threshold to match, and separates ranking from calibration. It should name the imbalance trap in ROC-AUC. Weak answers recite the definitions of precision and recall without saying which one to optimise or how prevalence changes the interpretation.

Sources