Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 12 of 27

Gradient descent

0 of 27 complete

Why it matters

Every model you call through an API was produced by this loop, and every fine-tuning job you run repeats it on your own data. The loop is small — compute a loss, compute its gradient, take a step — but the choices around it determine whether a training run costs a few GPU-hours or a few GPU-weeks, and whether it converges at all.

Even outside training, the vocabulary transfers. "Learning rate too high" describes the same failure in a fine-tuning run and in a classifier that oscillates. "The loss went down but the product got worse" is a statement about the difference between an objective and a goal. Knowing the mechanics is what lets you debug those statements instead of guessing.

Mental model: walking downhill in fog

You are standing on a hill in dense fog and want to reach the valley. You cannot see the landscape, but you can feel the slope under your feet. So you step in the steepest downhill direction, re-feel the slope, and repeat.

The gradient is that slope reading. The learning rate is how long your stride is. Too short and you crawl; too long and you step over the valley and up the other side.

The analogy breaks in high dimensions in three ways. First, the surface is not a bowl: it has plateaus, ridges and saddle points, and "downhill" can lead into a flat region where progress stalls. Second, the slope reading is noisy because it is estimated from a batch, not the whole dataset. Third, in thousands of dimensions there is almost always some direction that goes down, so the useful question is not "is this a local minimum" but "is the gradient small enough and the validation loss good enough to stop".

How it works

One rule, three step sizesInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Six synthetic iterates of gradient descent on a round quadratic, drawn in parameter space: θ₁ and θ₂, no unit. The loss is a function of the distance from the minimum, so the dashed rings are equal-loss contours and every point on a ring has the same loss. The minimum is the query at the origin. The factor per step belongs to a quadratic with curvature λ; a trained loss surface is not round, and batch noise, momentum and schedules are not in the figure.

What to notice. With ηλ = 0.5 each step halves the radius: 1.00, 0.50, 0.25. With ηλ = 2 the step lands on the opposite side of the same ring, so the loss is unchanged at 1.00 — the run neither converges nor diverges, it alternates. At ηλ = 2.5 the factor is 1.25 and the radius grows to 1.25, then 1.56. The gradient has the same size at every point on a ring; only its direction differs, and the step size, not the gradient, decides whether the run shrinks.

Text version

Six synthetic iterates of gradient descent in a two-parameter space, with the minimum at the origin marked as the query. The frame shows the converging chain at radii 0.25, 0.50 and 1.00 joined to the minimum, and the same start with an overshoot at the same radius 1.00 on the opposite side, plus two diverging iterates at 1.25 and 1.56 farther out. Dashed rings at 0.25, 0.50 and 1.00 are equal-loss contours. A slider sets how many iterates the rank panel lists, from one to six.

  1. Parameterise the model. Collect every learnable number into a vector θ\theta: weights, biases, embeddings.
  2. Define a loss. L(θ)L(\theta) is a single number summarising how wrong the model is on a batch. Mean squared error, cross-entropy and their variants are all differentiable by design.
  3. Compute the gradient. ∇L(θ)\nabla L(\theta) is the vector of partial derivatives: how much the loss changes per parameter. Backpropagation computes it exactly for a neural network in one backward pass.
  4. Take a step against the gradient. The gradient points uphill, so subtract it, scaled by the learning rate η\eta.
  5. Estimate the gradient on a batch. Full-batch descent is stable but slow; stochastic descent uses one example; minibatches of tens to thousands balance noise against hardware utilisation.
  6. Repeat over the data. One pass is an epoch. Shuffle between epochs so batch composition varies and the noise stays unbiased.
  7. Add state to the step. Momentum accumulates past gradients to push through flat regions and damp oscillation. Adaptive methods such as Adam scale the step per parameter using running estimates of the first and second moments.
  8. Schedule the step size. Start larger for speed, decay for stability; warm up briefly at the start of large runs to avoid early divergence.
  9. Monitor and stop. Track training loss, validation loss and gradient norm. Stop on validation performance, not on the training curve alone.

The math

The update rule, with learning rate η\eta:

θt+1=θt−η ∇θL(θt).\theta_{t+1} = \theta_t - \eta \, \nabla_\theta L(\theta_t).

For mean squared error over a batch of nn examples, the gradient has a convenient closed form:

L(θ)=1n∑i=1n(y^i−yi)2,∇θL=2n∑i=1n(y^i−yi)∇θy^i.L(\theta) = \frac{1}{n}\sum_{i=1}^{n}\left(\hat{y}_i - y_i\right)^2, \qquad \nabla_\theta L = \frac{2}{n}\sum_{i=1}^{n}\left(\hat{y}_i - y_i\right)\nabla_\theta \hat{y}_i.

Momentum replaces the raw gradient with an exponentially weighted average, which is why it keeps moving through short flat regions:

vt+1=βvt+∇L(θt),θt+1=θt−η vt+1.v_{t+1} = \beta v_t + \nabla L(\theta_t), \qquad \theta_{t+1} = \theta_t - \eta\, v_{t+1}.

A minimal implementation

import numpy as np
 
rng = np.random.default_rng(0)
X = rng.normal(size=(512, 3))
y = X @ np.array([2.0, -3.0, 0.5]) + 0.1 * rng.normal(size=512)
 
w = np.zeros(3)
lr, epochs, batch_size = 0.1, 60, 32
 
for epoch in range(epochs):
    order = rng.permutation(len(X))
    for start in range(0, len(X), batch_size):
        idx = order[start:start + batch_size]
        xb, yb = X[idx], y[idx]
        grad = 2 * xb.T @ (xb @ w - yb) / len(idx)   # d(MSE)/dw
        w -= lr * grad
    if epoch % 15 == 0:
        print(epoch, round(float(np.mean((X @ w - y) ** 2)), 4), np.round(w, 3))

Run it, then rerun with lr = 5.0. The loss grows instead of shrinking: that is divergence from an oversized step, and it is the first thing to check in any real training failure. Reset, standardise X, and the same learning rate converges faster because the surface is better conditioned.

Trade-offs

ChoiceGood forCostFails when
Full-batch descentSmall datasets, convex problemsOne pass over all data per stepDataset does not fit in memory
Stochastic (batch = 1)Streaming, very large dataNoisy gradients, poor hardware useBatch norm or hardware throughput matters
MinibatchDefault for neural networksBatch size becomes another hyperparameterBatch is so large that steps become too few per epoch
MomentumRavines, ill-conditioned surfacesOne extra hyperparameter to tuneMomentum is too high and the step overshoots repeatedly
Adam / AdamWSparse gradients, fast initial progressExtra memory per parameter, can generalise slightly worseWeight decay is implemented as L2 inside the gradient
Trade-off mapInteractive

Good for

Small datasets, convex problems

Cost

One pass over all data per step

Fails when

Dataset does not fit in memory

  1. Dataset does not fit in memory
  2. Batch norm or hardware throughput matters
  3. Batch is so large that steps become too few per epoch
  4. Momentum is too high and the step overshoots repeatedly
  5. Weight decay is implemented as L2 inside the gradient

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full-batch descent is good for Small datasets, convex problems, costs One pass over all data per step, and fails when Dataset does not fit in memory. Stochastic (batch = 1) is good for Streaming, very large data, costs Noisy gradients, poor hardware use, and fails when Batch norm or hardware throughput matters. Minibatch is good for Default for neural networks, costs Batch size becomes another hyperparameter, and fails when Batch is so large that steps become too few per epoch. Momentum is good for Ravines, ill-conditioned surfaces, costs One extra hyperparameter to tune, and fails when Momentum is too high and the step overshoots repeatedly. Adam / AdamW is good for Sparse gradients, fast initial progress, costs Extra memory per parameter, can generalise slightly worse, and fails when Weight decay is implemented as L2 inside the gradient

Failure modes

Divergence from a large learning rate. The loss jumps, then becomes nan. Debugging step: log the loss every step, restart at lr / 10, and clip the global gradient norm. A smooth curve that then explodes usually indicates a single bad batch rather than a bad rate.

A plateau caused by unscaled features. The loss falls briefly and then flattens well above target. Debugging step: standardise every input feature, check the condition number of the design matrix, and confirm the learning rate is not so small that progress is invisible per epoch.

Accumulated gradients. In frameworks that add to .grad by default, forgetting to zero them makes each step use the sum of all previous gradients; the loss becomes erratic and updates grow. Debugging step: print the gradient norm before and after step() and assert it returns to a small value.

Optimising the wrong split. Training loss keeps falling while validation loss rises. Debugging step: plot both on the same axes and stop at the minimum of validation loss; this is overfitting, not an optimiser fault.

In production

For fine-tuning, the practical controls are learning rate, batch size (often via gradient accumulation), warmup, gradient clipping and early stopping. Log all of them with the run, because a checkpoint without its optimiser settings is hard to reproduce.

Watch the gradient norm as a first-class metric. A sudden spike precedes most loss explosions, and sustained near-zero norms suggest dead units or a saturated objective. Both are cheaper to catch in the first hundred steps than after a day of compute.

Budget the cost honestly: optimiser state for Adam is two extra tensors per parameter, which is why full fine-tuning of a large model needs several times the memory of inference. Parameter-efficient methods change that arithmetic, not the loop.

Common misconceptions

People think: Gradient descent finds the global minimum of the loss.

Actually: It finds a point where the gradient is small. In high-dimensional non-convex problems that is usually a good-enough region, and saddle points are more common obstacles than local minima.

People think: A lower training loss means a better model.

Actually: Training loss measures fit to the data you trained on. Generalisation is measured on held-out data, and the two diverge once the model starts memorising.

A third: that Adam removes the need to tune the learning rate. It reduces sensitivity, but the rate still governs stability, and the default may be wrong for your batch size and objective.

Interview angle

Expect a question that starts concrete — "your fine-tune loss went to NaN, what happened?" — and then widens to learning rate, batch size, momentum and why Adam is not automatically better. A strong answer describes the update rule, names the failure each hyperparameter produces, and separates optimisation failure from data or objective failure. Compare notes with optimisers and loss functions, and see the training questions at /interview.

Sources