Why it matters
Fitting the training sample perfectly is easy and usually wrong. Regularisation narrows the set of functions the model can reach, or makes some of them more expensive, so the fit is forced toward patterns that hold outside the sample. The engineering difficulty is that the strength is data-dependent: the same dropout rate that stabilises a small dataset will cripple a large one, and the symptom of over-regularisation is a model that cannot fit even the training set.
Mental model
Regularisation is a tax on complexity: the model can still use a complicated explanation, but it pays for every unit of complexity it spends. The metaphor breaks because the tax is not on description length — it is on weight magnitude, activation noise or data transformations, and none of those map cleanly to Occam's razor.
How it works
- 1Establish the unregularised baseline and confirm it overfits before adding anything.
- 2Add the cheapest effective regulariser first: more data or label-preserving augmentation.
- 3Apply L2 weight decay to matrix weights, excluding biases and normalisation gains.
- 4Use L1 or elastic net when you need feature selection and can tolerate non-smooth gradients.
- 5Add dropout to wide or fully connected layers, and disable it at inference so the output is deterministic.
- 6Tune strength on validation performance and stop increasing when training loss stops falling.
Step 1 of 6
Establish the unregularised baseline and confirm it overfits before adding anything.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Establish the unregularised baseline and confirm it overfits before adding anything. 2. Add the cheapest effective regulariser first: more data or label-preserving augmentation. 3. Apply L2 weight decay to matrix weights, excluding biases and normalisation gains. 4. Use L1 or elastic net when you need feature selection and can tolerate non-smooth gradien… 5. Add dropout to wide or fully connected layers, and disable it at inference so the output … 6. Tune strength on validation performance and stop increasing when training loss stops fall…
- Establish the unregularised baseline and confirm it overfits before adding anything.
- Add the cheapest effective regulariser first: more data or label-preserving augmentation.
- Apply L2 weight decay to matrix weights, excluding biases and normalisation gains.
- Use L1 or elastic net when you need feature selection and can tolerate non-smooth gradients.
- Add dropout to wide or fully connected layers, and disable it at inference so the output is deterministic.
- Tune strength on validation performance and stop increasing when training loss stops falling.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| L2 weight decay | Smooth, stable, cheap | Does not produce sparse weights | You need feature selection |
| Dropout | Strong for wide layers, cheap to implement | Slower convergence, changes at inference | Sequence models with small batches |
| Data augmentation | Addresses the cause, often largest gain | Domain knowledge per modality, compute cost | Augmentation alters the label |
| Early stopping | No change to the model | Requires a reliable validation signal | Validation is noisy or leaked |
Good for
Smooth, stable, cheap
Cost
Does not produce sparse weights
Fails when
You need feature selection
- You need feature selection
- Sequence models with small batches
- Augmentation alters the label
- Validation is noisy or leaked
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: L2 weight decay is good for Smooth, stable, cheap, costs Does not produce sparse weights, and fails when You need feature selection. Dropout is good for Strong for wide layers, cheap to implement, costs Slower convergence, changes at inference, and fails when Sequence models with small batches. Data augmentation is good for Addresses the cause, often largest gain, costs Domain knowledge per modality, compute cost, and fails when Augmentation alters the label. Early stopping is good for No change to the model, costs Requires a reliable validation signal, and fails when Validation is noisy or leaked
Failure modes
- Regularisation left on at inference. Dropout remains active and predictions become noisy and weaker. Debug: compare eval-mode and train-mode outputs for the same batch; they should differ only if dropout is active.
- Weight decay on biases and norms. Small parameters are pulled to zero and the model loses calibration. Debug: list parameter groups and confirm which receive decay.
- Augmentation that changes the label. Cropping removes the object or flipping reverses text, so the model learns contradictory targets. Debug: visualise or decode a sample of augmented examples with their labels.
In production
Record every regularisation strength with the checkpoint, because inference code must reproduce the same eval-mode behaviour. Monitor the train-serving gap after deployment, since regularisation tuned for the training distribution can be too weak once traffic shifts.
Common misconceptions
People think: Regularisation always improves generalisation.
Actually: It reduces variance at the cost of bias; applied too strongly it prevents the model from learning the signal at all.
People think: Dropout is a universal default.
Actually: It helps wide dense layers most and can hurt transformers, small batches or sequence models where noise disrupts learned structure.
Interview angle
A strong answer distinguishes capacity control from data-side fixes, explains the mechanism of at least two regularisers, and names the symptom of over-regularisation. It should discuss where decay is applied. Weak answers list techniques without saying what each constrains or how you would detect too much of it.
Sources
- Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting — dropout mechanics and its regularising effect.
- Hastie, Tibshirani & Friedman, The Elements of Statistical Learning — shrinkage, lasso and ridge penalties.
- Bishop, Pattern Recognition and Machine Learning — regularisation as a prior over parameters.