Why it matters
Gradients are noisy and wildly different in scale across parameters: embedding rows for rare tokens barely move while output biases move constantly. A single global learning rate cannot serve both. Optimisers exist to reshape the gradient before applying it — smoothing it over steps, normalising it per parameter, or decaying weights separately — so one schedule can train the whole network without hand-tuning per layer.
Mental model
An optimiser is a driver steering through fog: the gradient is the slope under the wheels, momentum is the vehicle's inertia, and adaptive scaling is power steering that responds to resistance. The metaphor breaks because the terrain changes as the parameters move, so the vehicle alters the road it is driving on.
How it works
- 1Start with SGD plus momentum as the baseline when you can afford many epochs, since it often generalises well.
- 2Choose Adam or AdamW for transformers and sparse gradients, where per-parameter scaling saves tuning.
- 3Set the learning rate with a short sweep on a reduced run, then use warmup for the first few percent of steps.
- 4Apply weight decay as a separate parameter group, excluding biases and normalisation gains.
- 5Clip the global gradient norm before the step when training is unstable or sequences vary in length.
- 6Budget optimiser state: Adam keeps two extra tensors per parameter, so mixed precision and sharding change what fits.
Step 1 of 6
Start with SGD plus momentum as the baseline when you can afford many epochs, since it often generalises well.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Start with SGD plus momentum as the baseline when you can afford many epochs, since it of… 2. Choose Adam or AdamW for transformers and sparse gradients, where per-parameter scaling s… 3. Set the learning rate with a short sweep on a reduced run, then use warmup for the first … 4. Apply weight decay as a separate parameter group, excluding biases and normalisation gain… 5. Clip the global gradient norm before the step when training is unstable or sequences vary… 6. Budget optimiser state: Adam keeps two extra tensors per parameter, so mixed precision an…
- Start with SGD plus momentum as the baseline when you can afford many epochs, since it often generalises well.
- Choose Adam or AdamW for transformers and sparse gradients, where per-parameter scaling saves tuning.
- Set the learning rate with a short sweep on a reduced run, then use warmup for the first few percent of steps.
- Apply weight decay as a separate parameter group, excluding biases and normalisation gains.
- Clip the global gradient norm before the step when training is unstable or sequences vary in length.
- Budget optimiser state: Adam keeps two extra tensors per parameter, so mixed precision and sharding change what fits.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| SGD with momentum | Strong generalisation, low state memory | Sensitive to learning rate and conditioning | Gradients are sparse or scales vary widely |
| Adam | Fast early progress, robust to scaling | Two state tensors per parameter, can generalise slightly worse | Memory is tight or the run is long |
| AdamW | Correct decoupled weight decay | Same memory as Adam | You need the absolute lowest memory |
| Learning-rate schedules | Better final loss than a constant rate | Extra hyperparameters and a tuning cost | Runs are too short to benefit |
Good for
Strong generalisation, low state memory
Cost
Sensitive to learning rate and conditioning
Fails when
Gradients are sparse or scales vary widely
- Gradients are sparse or scales vary widely
- Memory is tight or the run is long
- You need the absolute lowest memory
- Runs are too short to benefit
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: SGD with momentum is good for Strong generalisation, low state memory, costs Sensitive to learning rate and conditioning, and fails when Gradients are sparse or scales vary widely. Adam is good for Fast early progress, robust to scaling, costs Two state tensors per parameter, can generalise slightly worse, and fails when Memory is tight or the run is long. AdamW is good for Correct decoupled weight decay, costs Same memory as Adam, and fails when You need the absolute lowest memory. Learning-rate schedules is good for Better final loss than a constant rate, costs Extra hyperparameters and a tuning cost, and fails when Runs are too short to benefit
Failure modes
- Weight decay applied through the gradient. The penalty is scaled by the adaptive denominator and large-gradient parameters are regularised less. Debug: compare the decay applied to a high-gradient and a low-gradient parameter with the same initial weight.
- Momentum without warmup. Early large steps destabilise a randomly initialised network. Debug: log loss for the first few hundred steps; a spike that recovers indicates missing warmup.
- State restored at the wrong dtype. Resuming from a checkpoint casts moment estimates and the update magnitude changes. Debug: compare the first ten updates after a restart against an uninterrupted run.
In production
Log the update-to-weight ratio per layer; a value that drifts by orders of magnitude means the optimiser is fighting the initialisation. Keep optimiser state in the same precision the run used, and include it in every checkpoint you intend to resume.
Common misconceptions
People think: Adam is strictly better than SGD.
Actually: It converges faster early and needs less tuning, but it costs memory and can generalise slightly worse on some vision tasks; the choice depends on budget and task.
People think: Weight decay and L2 regularisation are identical.
Actually: They coincide for plain SGD but diverge under adaptive methods, which is why decoupled decay exists.
Interview angle
A strong answer describes what each optimiser changes about the gradient, then connects that to memory and generalisation. It should explain decoupled weight decay and why warmup exists. Weak answers name Adam as the default without being able to say what state it keeps or when SGD would be preferable.
Sources
- Kingma & Ba, Adam: A Method for Stochastic Optimization — the adaptive moment estimates and bias correction.
- Loshchilov & Hutter, Decoupled Weight Decay Regularization — why weight decay should be separated from the adaptive step.
- Ruder, An overview of gradient descent optimization algorithms — a comparative account of SGD, momentum and adaptive methods.