Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 16 of 27

Overfitting

0 of 27 complete

Why it matters

Any flexible model can memorise its training examples, and the loss it minimises says nothing about examples it has not seen. The engineering problem is that the gap is invisible during training unless you hold data back and compare. Diagnosing overfitting correctly matters because the remedies differ: more data, less capacity, stronger regularisation or earlier stopping, and applying the wrong one wastes the run.

Mental model

Overfitting is studying the answer key instead of the subject: you can reproduce every past exam perfectly and fail a new question. The metaphor breaks because the model is not cheating deliberately — it is doing exactly what the loss asks, and the loss simply cannot distinguish signal from noise in the sample.

How it works

Mechanism, step by stepInteractive
  1. 1Hold out a validation split and plot training and validation loss on the same axes from the first epoch.
  2. 2Identify the epoch where validation loss starts to rise while training loss keeps falling.
  3. 3Check whether the gap shrinks with more data before reducing model capacity, since data is usually the cheaper fix.
  4. 4Add augmentation or regularisation and re-measure, watching for underfitting if you push too far.
  5. 5Use early stopping on validation loss with a patience window, and restore the best checkpoint.
  6. 6Compare the validation score with a second held-out set to confirm you have not tuned your way into it.

Step 1 of 6

Hold out a validation split and plot training and validation loss on the same axes from the first epoch.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Hold out a validation split and plot training and validation loss on the same axes from t… 2. Identify the epoch where validation loss starts to rise while training loss keeps falling. 3. Check whether the gap shrinks with more data before reducing model capacity, since data i… 4. Add augmentation or regularisation and re-measure, watching for underfitting if you push … 5. Use early stopping on validation loss with a patience window, and restore the best checkp… 6. Compare the validation score with a second held-out set to confirm you have not tuned you…

  1. Hold out a validation split and plot training and validation loss on the same axes from the first epoch.
  2. Identify the epoch where validation loss starts to rise while training loss keeps falling.
  3. Check whether the gap shrinks with more data before reducing model capacity, since data is usually the cheaper fix.
  4. Add augmentation or regularisation and re-measure, watching for underfitting if you push too far.
  5. Use early stopping on validation loss with a patience window, and restore the best checkpoint.
  6. Compare the validation score with a second held-out set to confirm you have not tuned your way into it.

Trade-offs

ChoiceGood forCostFails when
Early stoppingCheap, no architecture changeNeeds a clean validation split, stops before the schedule endsValidation is small or noisy
More dataAddresses the cause directlyCollection cost and possible distribution driftThe label budget is fixed
Smaller modelLower variance, cheaper servingMay underfit the real patternThe mapping is genuinely complex
Heavy augmentationEffective data multiplierCan distort the label semanticsAugmentations change what the target means
Trade-off mapInteractive

Good for

Cheap, no architecture change

Cost

Needs a clean validation split, stops before the schedule ends

Fails when

Validation is small or noisy

  1. Validation is small or noisy
  2. The label budget is fixed
  3. The mapping is genuinely complex
  4. Augmentations change what the target means

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Early stopping is good for Cheap, no architecture change, costs Needs a clean validation split, stops before the schedule ends, and fails when Validation is small or noisy. More data is good for Addresses the cause directly, costs Collection cost and possible distribution drift, and fails when The label budget is fixed. Smaller model is good for Lower variance, cheaper serving, costs May underfit the real pattern, and fails when The mapping is genuinely complex. Heavy augmentation is good for Effective data multiplier, costs Can distort the label semantics, and fails when Augmentations change what the target means

Failure modes

  • Validation loss never rises. You conclude there is no overfitting because the curve is monotone, but the split is too easy or leaked. Debug: check for duplicate examples across splits and for a validation set drawn from the same narrow source.
  • Early stopping on the test set. The test score becomes a tuned quantity and overstates deployment performance. Debug: confirm the stopping signal comes from validation, never from test.
  • Memorisation of rare classes. Overall loss looks fine while the rare class is predicted from memorised examples. Debug: report per-class validation metrics and inspect the rare-class errors.

In production

Track the train-serving gap continuously, since deployment drift can make a model behave as if it were overfit to an old sample. Keep a rolling evaluation window and alert when the gap widens rather than when a single score dips.

Common misconceptions

People think: A model with low training error is overfit.

Actually: Low training error is necessary for a good model; overfitting is the gap between training and held-out performance, not training performance itself.

People think: Overfitting only happens with too many parameters.

Actually: It happens whenever the model can latch onto sample-specific noise, including with small models trained too long or on leaked features.

Interview angle

A strong answer defines overfitting by the generalisation gap, then ranks remedies by cause: data, capacity, regularisation, stopping. It should mention that validation can be overfit through tuning. Weak answers equate overfitting with a large model and cannot say what the learning curves would look like.

Sources