Why it matters
Length is not a single concept: it can mean total mass, Euclidean distance or worst coordinate, and each definition leads to a different penalty and a different notion of closeness. Regularisation uses L1 or L2 to push weights toward zero in different ways, gradient clipping uses a norm to bound step size, and retrieval uses normalisation to stop magnitude from dominating similarity. Picking the wrong norm changes behaviour, not just scale.
Mental model
A norm is a way of asking how big a vector is: L1 adds up the luggage, L2 measures the straight-line distance to the origin, max norm reports the heaviest single bag. The metaphor breaks because these measures can disagree sharply — a vector with one huge coordinate is long under max norm and only moderate under L1.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Eight synthetic vectors with L1 norm exactly 1.00, plotted by their two coordinates. The four on the axes are the corners of the L1 ball; the four at (±0.5, ±0.5) are its edge midpoints. The dashed rings are L2 iso-norm circles at 0.71 and 1.00, centred on the queried origin. The max-norm ball is a square and is not drawn, and the third dimension and above are not in the figure.
What to notice. Every point here has the same L1 total, 1.00, yet the L2 lengths are 1.00 at the axis points and 0.71 at the edge midpoints. The corner of the L1 ball touches the L2 circle; its edge sags inside it. So 'which vector is bigger' has two answers: L1 calls all eight equal, and L2 orders them 1.00 against 0.71. Max norm would call the axis points 1.00 and the midpoints 0.50 — the same split as L2 here, by a different measure. A norm is a shape as much as a number.
Text version
Eight synthetic vectors arranged around the origin on a two-coordinate grid. Four sit on the axes at (1,0), (0,1), (−1,0) and (0,−1), and four at (0.5,0.5), (−0.5,0.5), (−0.5,−0.5) and (0.5,−0.5). The query marker is the origin, so each distance in the rank panel is that vector's L2 length: the four edge midpoints print 0.71 and the four axis points print 1.00, in that order. Dashed rings at 0.71 and 1.00 pass through the midpoints and the axis points respectively. Two step controls first show the points alone, then the origin and the iso-norm rings.
- Decide what 'big' should mean for the problem: total deviation, Euclidean distance or worst-case coordinate.
- Compute L2 as the square root of the sum of squares, or use a library routine that handles overflow safely.
- Compute L1 as the sum of absolute values when sparsity is the goal.
- Normalise by dividing by the L2 norm when comparing directions, adding epsilon to avoid division by zero.
- Use the norm of the gradient for clipping, since it bounds the update across all parameters at once.
- Compare norms across batches only after checking that dimensionality and scaling are unchanged.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| L2 norm | Smooth, differentiable, rotation-invariant | Sensitive to outliers and large coordinates | A few dimensions dominate the data |
| L1 norm | Robust to outliers, induces sparsity | Non-differentiable at zero, less stable optimisation | You need smooth gradients near the origin |
| Max norm | Bounds the worst coordinate, cheap | Ignores the rest of the vector | Total magnitude matters |
| Unit normalisation | Comparable directions, bounded scores | Discards magnitude and amplifies noise in short vectors | Short or near-zero vectors are common |
Good for
Smooth, differentiable, rotation-invariant
Cost
Sensitive to outliers and large coordinates
Fails when
A few dimensions dominate the data
- A few dimensions dominate the data
- You need smooth gradients near the origin
- Total magnitude matters
- Short or near-zero vectors are common
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: L2 norm is good for Smooth, differentiable, rotation-invariant, costs Sensitive to outliers and large coordinates, and fails when A few dimensions dominate the data. L1 norm is good for Robust to outliers, induces sparsity, costs Non-differentiable at zero, less stable optimisation, and fails when You need smooth gradients near the origin. Max norm is good for Bounds the worst coordinate, cheap, costs Ignores the rest of the vector, and fails when Total magnitude matters. Unit normalisation is good for Comparable directions, bounded scores, costs Discards magnitude and amplifies noise in short vectors, and fails when Short or near-zero vectors are common
Failure modes
- Division by a near-zero norm. Normalisation explodes and produces huge or NaN values for zero vectors. Debug: count vectors whose norm is below epsilon in the batch before normalising.
- Norm computed after clipping. The reported gradient norm always looks acceptable because it is measured post-clip. Debug: log the norm before and after clipping separately.
- Mixed norms in one pipeline. Training penalises L2 while evaluation ranks by L1 distance, so the objective and the metric disagree. Debug: trace the metric implementation and confirm which norm it uses.
In production
Monitor the distribution of gradient norms and embedding norms, not just their means, since a rising tail predicts instability or ranking drift. Cap norms explicitly at write time so a single pathological vector cannot dominate an index.
Common misconceptions
People think: All norms measure the same thing up to a constant.
Actually: They can rank vectors differently; a vector can be largest under one norm and moderate under another, which is why the choice changes sparsity and robustness.
People think: Normalising vectors is always harmless preprocessing.
Actually: It removes magnitude, which may be signal, and it amplifies direction noise when the original vector is close to zero.
Interview angle
A strong answer defines the three common norms, connects each to a use — regularisation, clipping, similarity — and explains what unit normalisation discards. It should mention the zero-norm edge case. Weak answers recite formulas without saying when L1 is preferable to L2 or why clipping reports the norm it does.
Sources
- NumPy, numpy.linalg.norm — ord parameters and their definitions.
- 3Blue1Brown, Essence of linear algebra — geometric length and distance.
- Murphy, Probabilistic Machine Learning: An Introduction — norm penalties and their role in estimation.