Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 13 of 27

Optimisers

0 of 27 complete

Why it matters

Gradients are noisy and wildly different in scale across parameters: embedding rows for rare tokens barely move while output biases move constantly. A single global learning rate cannot serve both. Optimisers exist to reshape the gradient before applying it — smoothing it over steps, normalising it per parameter, or decaying weights separately — so one schedule can train the whole network without hand-tuning per layer.

Mental model

An optimiser is a driver steering through fog: the gradient is the slope under the wheels, momentum is the vehicle's inertia, and adaptive scaling is power steering that responds to resistance. The metaphor breaks because the terrain changes as the parameters move, so the vehicle alters the road it is driving on.

How it works

Mechanism, step by stepInteractive
  1. 1Start with SGD plus momentum as the baseline when you can afford many epochs, since it often generalises well.
  2. 2Choose Adam or AdamW for transformers and sparse gradients, where per-parameter scaling saves tuning.
  3. 3Set the learning rate with a short sweep on a reduced run, then use warmup for the first few percent of steps.
  4. 4Apply weight decay as a separate parameter group, excluding biases and normalisation gains.
  5. 5Clip the global gradient norm before the step when training is unstable or sequences vary in length.
  6. 6Budget optimiser state: Adam keeps two extra tensors per parameter, so mixed precision and sharding change what fits.

Step 1 of 6

Start with SGD plus momentum as the baseline when you can afford many epochs, since it often generalises well.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Start with SGD plus momentum as the baseline when you can afford many epochs, since it of… 2. Choose Adam or AdamW for transformers and sparse gradients, where per-parameter scaling s… 3. Set the learning rate with a short sweep on a reduced run, then use warmup for the first … 4. Apply weight decay as a separate parameter group, excluding biases and normalisation gain… 5. Clip the global gradient norm before the step when training is unstable or sequences vary… 6. Budget optimiser state: Adam keeps two extra tensors per parameter, so mixed precision an…

  1. Start with SGD plus momentum as the baseline when you can afford many epochs, since it often generalises well.
  2. Choose Adam or AdamW for transformers and sparse gradients, where per-parameter scaling saves tuning.
  3. Set the learning rate with a short sweep on a reduced run, then use warmup for the first few percent of steps.
  4. Apply weight decay as a separate parameter group, excluding biases and normalisation gains.
  5. Clip the global gradient norm before the step when training is unstable or sequences vary in length.
  6. Budget optimiser state: Adam keeps two extra tensors per parameter, so mixed precision and sharding change what fits.

Trade-offs

ChoiceGood forCostFails when
SGD with momentumStrong generalisation, low state memorySensitive to learning rate and conditioningGradients are sparse or scales vary widely
AdamFast early progress, robust to scalingTwo state tensors per parameter, can generalise slightly worseMemory is tight or the run is long
AdamWCorrect decoupled weight decaySame memory as AdamYou need the absolute lowest memory
Learning-rate schedulesBetter final loss than a constant rateExtra hyperparameters and a tuning costRuns are too short to benefit
Trade-off mapInteractive

Good for

Strong generalisation, low state memory

Cost

Sensitive to learning rate and conditioning

Fails when

Gradients are sparse or scales vary widely

  1. Gradients are sparse or scales vary widely
  2. Memory is tight or the run is long
  3. You need the absolute lowest memory
  4. Runs are too short to benefit

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: SGD with momentum is good for Strong generalisation, low state memory, costs Sensitive to learning rate and conditioning, and fails when Gradients are sparse or scales vary widely. Adam is good for Fast early progress, robust to scaling, costs Two state tensors per parameter, can generalise slightly worse, and fails when Memory is tight or the run is long. AdamW is good for Correct decoupled weight decay, costs Same memory as Adam, and fails when You need the absolute lowest memory. Learning-rate schedules is good for Better final loss than a constant rate, costs Extra hyperparameters and a tuning cost, and fails when Runs are too short to benefit

Failure modes

  • Weight decay applied through the gradient. The penalty is scaled by the adaptive denominator and large-gradient parameters are regularised less. Debug: compare the decay applied to a high-gradient and a low-gradient parameter with the same initial weight.
  • Momentum without warmup. Early large steps destabilise a randomly initialised network. Debug: log loss for the first few hundred steps; a spike that recovers indicates missing warmup.
  • State restored at the wrong dtype. Resuming from a checkpoint casts moment estimates and the update magnitude changes. Debug: compare the first ten updates after a restart against an uninterrupted run.

In production

Log the update-to-weight ratio per layer; a value that drifts by orders of magnitude means the optimiser is fighting the initialisation. Keep optimiser state in the same precision the run used, and include it in every checkpoint you intend to resume.

Common misconceptions

People think: Adam is strictly better than SGD.

Actually: It converges faster early and needs less tuning, but it costs memory and can generalise slightly worse on some vision tasks; the choice depends on budget and task.

People think: Weight decay and L2 regularisation are identical.

Actually: They coincide for plain SGD but diverge under adaptive methods, which is why decoupled decay exists.

Interview angle

A strong answer describes what each optimiser changes about the gradient, then connects that to memory and generalisation. It should explain decoupled weight decay and why warmup exists. Weak answers name Adam as the default without being able to say what state it keeps or when SGD would be preferable.

Sources