Why it matters
Every dense layer, embedding lookup and attention projection is a matrix multiply, and most shape errors are matrix-shape errors. Beyond correctness, the dimensions decide parameter count and FLOPs, while rank decides how much a layer can represent. Treating a weight matrix as a table of numbers rather than a linear map makes it hard to reason about low-rank adaptation, projections and the difference between a transpose and a reshape.
Mental model
A matrix is a machine that takes a vector in and emits a vector out, with a fixed recipe for mixing the input coordinates. The metaphor breaks because the machine is linear: it cannot create a new direction that was not already in its columns, so its expressive power is bounded by its rank.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Six outputs of a synthetic 2×2 matrix, M = [[0.5, 1.0], [0.25, 0.5]], plotted in output coordinates with the input that produced each one in its label. The matrix has rank one, so every output lies on one line through the origin, and the reference vector is the query. The axes are the two output coordinates, no unit. The multiplication is not animated and the input plane is not drawn; only its images are.
What to notice. Every output is a multiple of the first column, (0.5, 0.25), so the whole plane is mapped onto one line. Cosine against the reference is 1.00 for outputs in the positive direction and −1.00 for the two negatives, with nothing in between. The dot product ranks by length instead: 1.13, 0.84, 0.56, 0.00, −0.56, −1.13. The non-zero input (2, −1) maps exactly to the origin — it is in the null space — so two different inputs share one output and the map is not invertible. A full-rank map would fill the plane, and is not drawn.
Text version
A scatter of six outputs of a synthetic rank-one matrix in two output coordinates, with the reference vector at 0.90, 0.45 marked as the query. The default frame uses euclidean distance and lists all six: M(0,1) at 0.11, M(0.5,0.5) at 0.17, M(1,0) at 0.45, the zero output of the non-zero input (2, −1) at 1.01, and the two negative outputs at 1.57 and 2.12. Every marker lies on the line y = x/2. A metric control switches to cosine, where the three positive outputs tie at 1.00, the two negatives score −1.00 and the zero output scores 0.00, and to the dot product, which ranks by length.
- Read the shape as (out features, in features) for a weight matrix, so the multiply consumes the input's last dimension.
- Check the inner dimensions match, and predict the output shape before running the code.
- Compute rank when a layer seems unable to fit: a rank-deficient weight map ignores part of its input space.
- Use transposes to align dimensions, remembering that a transpose changes strides rather than moving data.
- Batch the multiply by stacking examples in a leading dimension, so one call replaces a Python loop.
- Express multi-term contractions with einsum when the index bookkeeping is unclear, then check the result against explicit matmuls.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Dense matrix | Full-rank expressiveness, hardware-friendly | Parameters scale with both dimensions | The layer is huge and mostly redundant |
| Low-rank factorisation | Far fewer parameters, cheaper adaptation | Restricted to a subspace of the full map | The task needs a full-rank transformation |
| Batch matmul | High device utilisation | Memory for intermediate products | Batch or head dimensions are very large |
| Sparse matrix | Memory proportional to non-zeros | Irregular access, weaker hardware support | The matrix is mostly dense in practice |
Good for
Full-rank expressiveness, hardware-friendly
Cost
Parameters scale with both dimensions
Fails when
The layer is huge and mostly redundant
- The layer is huge and mostly redundant
- The task needs a full-rank transformation
- Batch or head dimensions are very large
- The matrix is mostly dense in practice
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Dense matrix is good for Full-rank expressiveness, hardware-friendly, costs Parameters scale with both dimensions, and fails when The layer is huge and mostly redundant. Low-rank factorisation is good for Far fewer parameters, cheaper adaptation, costs Restricted to a subspace of the full map, and fails when The task needs a full-rank transformation. Batch matmul is good for High device utilisation, costs Memory for intermediate products, and fails when Batch or head dimensions are very large. Sparse matrix is good for Memory proportional to non-zeros, costs Irregular access, weaker hardware support, and fails when The matrix is mostly dense in practice
Failure modes
- Silent broadcasting in matmul. A missing dimension is treated as a batch of one and the result has an unexpected shape. Debug: print both operand shapes and the output shape, then compare against a hand-computed expectation.
- Rank collapse. A product of matrices loses rank and the layer cannot distinguish inputs it should. Debug: compute the singular values of the weight and check how many are effectively zero.
- Transpose mistaken for reshape. The values land in the wrong positions while the shape looks correct. Debug: multiply by a one-hot vector and check which output coordinate becomes non-zero.
In production
Assert weight shapes against the config at load time, because a checkpoint with a transposed matrix can still load and produce plausible but degraded output. When adapting a model with low-rank updates, log the effective rank so a collapsed adapter is visible.
Common misconceptions
People think: A matrix is just a 2D array.
Actually: The array is the storage; the mathematical object is a linear map, and properties like rank and null space describe what it can and cannot represent.
People think: More parameters in a matrix always mean more capacity.
Actually: Capacity depends on rank and on training signal; a large matrix trained on little data can behave like a much smaller one.
Interview angle
A strong answer treats a weight matrix as a linear map, reads shapes fluently, and connects rank to representational limits. It should distinguish transpose from reshape and explain batching. Weak answers can multiply matrices on paper but cannot say what a rank-deficient weight implies or why shape errors dominate implementation bugs.
Sources
- MIT OpenCourseWare, 18.06 Linear Algebra — linear maps, rank and matrix multiplication.
- 3Blue1Brown, Essence of linear algebra — visual intuition for matrices as transformations.
- NumPy, Linear algebra — matmul, rank and decomposition routines.