Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 11 of 27

Derivatives

0 of 27 complete

Why it matters

Training needs to know which direction to move each parameter to reduce loss, and for billions of parameters that direction must be computed cheaply. Derivatives provide it locally: the gradient is the direction of steepest ascent, so its negative is the update direction. The chain rule makes the computation tractable by composing per-operation derivatives, and the local nature of the derivative is also why step size matters so much.

Mental model

A derivative is the slope of a hill under your feet: it tells you which way is up right now, not where the summit is. The metaphor breaks because the terrain is high-dimensional and not smooth everywhere — a slope exists at a point but says little about the distance to a better valley.

How it works

A slope is a secant between samplesInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Seven samples of a synthetic f(x) = x², placed at their own coordinates, with the evaluation point at x = 0.2 marked. Every difference quotient is a slope between two of these points, so k chooses how many of the nearest samples are joined to the evaluation point. The axes are the input and the output, both without units. The tangent is a limit the figure only approaches: it is stated in this caption, not drawn.

What to notice. The nearest sample is 0.05 away in x and gives a slope of 0.45 against a true derivative of 0.40 at x = 0.2. The second sample, on the other side, gives 0.30, so the two nearest bracket the tangent. The third, 0.20 away, gives 0.60 — the estimate is worse rather than better, because a wider stencil follows the curve's bend. The dashed ring is 0.25 in the plane of the plot: a stencil radius, not a Δx, though on this shallow curve the sample order still follows |Δx|.

Text version

A scatter of seven synthetic samples on the curve f(x) = x², with the point at x = 0.2 marked as the evaluation point. The default frame uses k = 3 and joins the three nearest samples: x = 0.25 at 0.05, x = 0.10 at 0.10 and x = 0.40 at 0.23, the first three inside or near a dashed ring of radius 0.25. Their secants have slopes 0.45, 0.30 and 0.60, against a tangent of 0.40. A slider sets k from one to five; each slope is read from the plotted coordinates as rise over run.

  1. Write the function as a composition of simple operations whose derivatives are known.
  2. Differentiate the outermost operation, then multiply by the derivative of its input, applying the chain rule inward.
  3. For multivariate functions, take partial derivatives one input at a time while holding the rest constant.
  4. Assemble the partials into the gradient vector, matching each entry to a parameter.
  5. Check the result with a central difference on a few coordinates, using a small step and float64.
  6. Use the gradient's norm to judge scale and its direction to choose the update sign.

Trade-offs

ChoiceGood forCostFails when
Analytic gradientExact and cheap to evaluateRequires a correct derivation per operationThe function has branches or discontinuities
Numerical differenceTrivial to implement, good as a checkCost grows with parameter count, sensitive to step sizeThe model has more than a handful of parameters
Automatic differentiationExact and scalable, no manual derivationGraph memory and framework constraintsThe operation has no defined gradient
Subgradient at non-smooth pointsAllows training ReLU-like functionsNot a true direction of descent everywhereThe loss is highly non-smooth
Trade-off mapInteractive

Good for

Exact and cheap to evaluate

Cost

Requires a correct derivation per operation

Fails when

The function has branches or discontinuities

  1. The function has branches or discontinuities
  2. The model has more than a handful of parameters
  3. The operation has no defined gradient
  4. The loss is highly non-smooth

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Analytic gradient is good for Exact and cheap to evaluate, costs Requires a correct derivation per operation, and fails when The function has branches or discontinuities. Numerical difference is good for Trivial to implement, good as a check, costs Cost grows with parameter count, sensitive to step size, and fails when The model has more than a handful of parameters. Automatic differentiation is good for Exact and scalable, no manual derivation, costs Graph memory and framework constraints, and fails when The operation has no defined gradient. Subgradient at non-smooth points is good for Allows training ReLU-like functions, costs Not a true direction of descent everywhere, and fails when The loss is highly non-smooth

Failure modes

  • Step size too large in a numerical check. The difference quotient disagrees with the analytic gradient because the function curves within the step. Debug: halve the step and confirm the error falls before rising again from rounding.
  • Gradient sign flipped. Loss increases steadily after each step while the code looks correct. Debug: compare one parameter update against a numerical derivative in that coordinate.
  • Discontinuity treated as differentiable. A hard threshold or argmax blocks gradient flow and the parameter never updates. Debug: check whether the parameter's gradient is exactly zero across batches.

In production

Keep a gradient check for any custom operation and run it in CI on a small input, because a wrong derivative trains slowly rather than failing loudly. Monitor gradient norms per layer to catch the scale problems that a local slope check cannot.

Common misconceptions

People think: The gradient points at the minimum.

Actually: It points at the direction of steepest increase locally; the minimum may be far away, and the step size decides whether you get closer.

People think: A zero gradient means you have found the optimum.

Actually: It can be a saddle point, a plateau or a dead unit, and in high dimensions saddles are far more common than minima.

Interview angle

A strong answer defines the derivative as a local limit, extends it to partials and the gradient, and shows how the chain rule yields backpropagation. It should mention numerical checking and non-smooth points. Weak answers differentiate a polynomial but cannot explain why the gradient is the update direction or what a zero gradient might mean.

Sources