Why it matters
Every model you call through an API was produced by this loop, and every fine-tuning job you run repeats it on your own data. The loop is small — compute a loss, compute its gradient, take a step — but the choices around it determine whether a training run costs a few GPU-hours or a few GPU-weeks, and whether it converges at all.
Even outside training, the vocabulary transfers. "Learning rate too high" describes the same failure in a fine-tuning run and in a classifier that oscillates. "The loss went down but the product got worse" is a statement about the difference between an objective and a goal. Knowing the mechanics is what lets you debug those statements instead of guessing.
Mental model: walking downhill in fog
You are standing on a hill in dense fog and want to reach the valley. You cannot see the landscape, but you can feel the slope under your feet. So you step in the steepest downhill direction, re-feel the slope, and repeat.
The gradient is that slope reading. The learning rate is how long your stride is. Too short and you crawl; too long and you step over the valley and up the other side.
The analogy breaks in high dimensions in three ways. First, the surface is not a bowl: it has plateaus, ridges and saddle points, and "downhill" can lead into a flat region where progress stalls. Second, the slope reading is noisy because it is estimated from a batch, not the whole dataset. Third, in thousands of dimensions there is almost always some direction that goes down, so the useful question is not "is this a local minimum" but "is the gradient small enough and the validation loss good enough to stop".
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Six synthetic iterates of gradient descent on a round quadratic, drawn in parameter space: θ₁ and θ₂, no unit. The loss is a function of the distance from the minimum, so the dashed rings are equal-loss contours and every point on a ring has the same loss. The minimum is the query at the origin. The factor per step belongs to a quadratic with curvature λ; a trained loss surface is not round, and batch noise, momentum and schedules are not in the figure.
What to notice. With ηλ = 0.5 each step halves the radius: 1.00, 0.50, 0.25. With ηλ = 2 the step lands on the opposite side of the same ring, so the loss is unchanged at 1.00 — the run neither converges nor diverges, it alternates. At ηλ = 2.5 the factor is 1.25 and the radius grows to 1.25, then 1.56. The gradient has the same size at every point on a ring; only its direction differs, and the step size, not the gradient, decides whether the run shrinks.
Text version
Six synthetic iterates of gradient descent in a two-parameter space, with the minimum at the origin marked as the query. The frame shows the converging chain at radii 0.25, 0.50 and 1.00 joined to the minimum, and the same start with an overshoot at the same radius 1.00 on the opposite side, plus two diverging iterates at 1.25 and 1.56 farther out. Dashed rings at 0.25, 0.50 and 1.00 are equal-loss contours. A slider sets how many iterates the rank panel lists, from one to six.
- Parameterise the model. Collect every learnable number into a vector : weights, biases, embeddings.
- Define a loss. is a single number summarising how wrong the model is on a batch. Mean squared error, cross-entropy and their variants are all differentiable by design.
- Compute the gradient. is the vector of partial derivatives: how much the loss changes per parameter. Backpropagation computes it exactly for a neural network in one backward pass.
- Take a step against the gradient. The gradient points uphill, so subtract it, scaled by the learning rate .
- Estimate the gradient on a batch. Full-batch descent is stable but slow; stochastic descent uses one example; minibatches of tens to thousands balance noise against hardware utilisation.
- Repeat over the data. One pass is an epoch. Shuffle between epochs so batch composition varies and the noise stays unbiased.
- Add state to the step. Momentum accumulates past gradients to push through flat regions and damp oscillation. Adaptive methods such as Adam scale the step per parameter using running estimates of the first and second moments.
- Schedule the step size. Start larger for speed, decay for stability; warm up briefly at the start of large runs to avoid early divergence.
- Monitor and stop. Track training loss, validation loss and gradient norm. Stop on validation performance, not on the training curve alone.
The math
The update rule, with learning rate :
For mean squared error over a batch of examples, the gradient has a convenient closed form:
Momentum replaces the raw gradient with an exponentially weighted average, which is why it keeps moving through short flat regions:
A minimal implementation
import numpy as np
rng = np.random.default_rng(0)
X = rng.normal(size=(512, 3))
y = X @ np.array([2.0, -3.0, 0.5]) + 0.1 * rng.normal(size=512)
w = np.zeros(3)
lr, epochs, batch_size = 0.1, 60, 32
for epoch in range(epochs):
order = rng.permutation(len(X))
for start in range(0, len(X), batch_size):
idx = order[start:start + batch_size]
xb, yb = X[idx], y[idx]
grad = 2 * xb.T @ (xb @ w - yb) / len(idx) # d(MSE)/dw
w -= lr * grad
if epoch % 15 == 0:
print(epoch, round(float(np.mean((X @ w - y) ** 2)), 4), np.round(w, 3))Run it, then rerun with lr = 5.0. The loss grows instead of shrinking: that is divergence from an oversized step, and it is the first thing to check in any real training failure. Reset, standardise X, and the same learning rate converges faster because the surface is better conditioned.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Full-batch descent | Small datasets, convex problems | One pass over all data per step | Dataset does not fit in memory |
| Stochastic (batch = 1) | Streaming, very large data | Noisy gradients, poor hardware use | Batch norm or hardware throughput matters |
| Minibatch | Default for neural networks | Batch size becomes another hyperparameter | Batch is so large that steps become too few per epoch |
| Momentum | Ravines, ill-conditioned surfaces | One extra hyperparameter to tune | Momentum is too high and the step overshoots repeatedly |
| Adam / AdamW | Sparse gradients, fast initial progress | Extra memory per parameter, can generalise slightly worse | Weight decay is implemented as L2 inside the gradient |
Good for
Small datasets, convex problems
Cost
One pass over all data per step
Fails when
Dataset does not fit in memory
- Dataset does not fit in memory
- Batch norm or hardware throughput matters
- Batch is so large that steps become too few per epoch
- Momentum is too high and the step overshoots repeatedly
- Weight decay is implemented as L2 inside the gradient
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Full-batch descent is good for Small datasets, convex problems, costs One pass over all data per step, and fails when Dataset does not fit in memory. Stochastic (batch = 1) is good for Streaming, very large data, costs Noisy gradients, poor hardware use, and fails when Batch norm or hardware throughput matters. Minibatch is good for Default for neural networks, costs Batch size becomes another hyperparameter, and fails when Batch is so large that steps become too few per epoch. Momentum is good for Ravines, ill-conditioned surfaces, costs One extra hyperparameter to tune, and fails when Momentum is too high and the step overshoots repeatedly. Adam / AdamW is good for Sparse gradients, fast initial progress, costs Extra memory per parameter, can generalise slightly worse, and fails when Weight decay is implemented as L2 inside the gradient
Failure modes
Divergence from a large learning rate. The loss jumps, then becomes nan. Debugging step: log the loss every step, restart at lr / 10, and clip the global gradient norm. A smooth curve that then explodes usually indicates a single bad batch rather than a bad rate.
A plateau caused by unscaled features. The loss falls briefly and then flattens well above target. Debugging step: standardise every input feature, check the condition number of the design matrix, and confirm the learning rate is not so small that progress is invisible per epoch.
Accumulated gradients. In frameworks that add to .grad by default, forgetting to zero them makes each step use the sum of all previous gradients; the loss becomes erratic and updates grow. Debugging step: print the gradient norm before and after step() and assert it returns to a small value.
Optimising the wrong split. Training loss keeps falling while validation loss rises. Debugging step: plot both on the same axes and stop at the minimum of validation loss; this is overfitting, not an optimiser fault.
In production
For fine-tuning, the practical controls are learning rate, batch size (often via gradient accumulation), warmup, gradient clipping and early stopping. Log all of them with the run, because a checkpoint without its optimiser settings is hard to reproduce.
Watch the gradient norm as a first-class metric. A sudden spike precedes most loss explosions, and sustained near-zero norms suggest dead units or a saturated objective. Both are cheaper to catch in the first hundred steps than after a day of compute.
Budget the cost honestly: optimiser state for Adam is two extra tensors per parameter, which is why full fine-tuning of a large model needs several times the memory of inference. Parameter-efficient methods change that arithmetic, not the loop.
Common misconceptions
People think: Gradient descent finds the global minimum of the loss.
Actually: It finds a point where the gradient is small. In high-dimensional non-convex problems that is usually a good-enough region, and saddle points are more common obstacles than local minima.
People think: A lower training loss means a better model.
Actually: Training loss measures fit to the data you trained on. Generalisation is measured on held-out data, and the two diverge once the model starts memorising.
A third: that Adam removes the need to tune the learning rate. It reduces sensitivity, but the rate still governs stability, and the default may be wrong for your batch size and objective.
Interview angle
Expect a question that starts concrete — "your fine-tune loss went to NaN, what happened?" — and then widens to learning rate, batch size, momentum and why Adam is not automatically better. A strong answer describes the update rule, names the failure each hyperparameter produces, and separates optimisation failure from data or objective failure. Compare notes with optimisers and loss functions, and see the training questions at /interview.
Sources
- Stanford CS231n, Optimization: Stochastic Gradient Descent — update rules, momentum and learning-rate intuition: https://cs231n.github.io/optimization-1/
- Kingma & Ba, Adam: A Method for Stochastic Optimization — the adaptive moment estimate used by most fine-tuning defaults: https://arxiv.org/abs/1412.6980
- Goh, Why Momentum Really Works, Distill — a visual treatment of conditioning and momentum: https://distill.pub/2017/momentum/
- Dauphin et al., Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization — why saddle points, not local minima, dominate: https://arxiv.org/abs/1406.2572