Skip to content
AI Engineer Guide.

concept · basics · intermediate · concept 2 of 27

Matrices

0 of 27 complete

Why it matters

Every dense layer, embedding lookup and attention projection is a matrix multiply, and most shape errors are matrix-shape errors. Beyond correctness, the dimensions decide parameter count and FLOPs, while rank decides how much a layer can represent. Treating a weight matrix as a table of numbers rather than a linear map makes it hard to reason about low-rank adaptation, projections and the difference between a transpose and a reshape.

Mental model

A matrix is a machine that takes a vector in and emits a vector out, with a fixed recipe for mixing the input coordinates. The metaphor breaks because the machine is linear: it cannot create a new direction that was not already in its columns, so its expressive power is bounded by its rank.

How it works

Rank one: the plane lands on a lineInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Six outputs of a synthetic 2×2 matrix, M = [[0.5, 1.0], [0.25, 0.5]], plotted in output coordinates with the input that produced each one in its label. The matrix has rank one, so every output lies on one line through the origin, and the reference vector is the query. The axes are the two output coordinates, no unit. The multiplication is not animated and the input plane is not drawn; only its images are.

What to notice. Every output is a multiple of the first column, (0.5, 0.25), so the whole plane is mapped onto one line. Cosine against the reference is 1.00 for outputs in the positive direction and −1.00 for the two negatives, with nothing in between. The dot product ranks by length instead: 1.13, 0.84, 0.56, 0.00, −0.56, −1.13. The non-zero input (2, −1) maps exactly to the origin — it is in the null space — so two different inputs share one output and the map is not invertible. A full-rank map would fill the plane, and is not drawn.

Text version

A scatter of six outputs of a synthetic rank-one matrix in two output coordinates, with the reference vector at 0.90, 0.45 marked as the query. The default frame uses euclidean distance and lists all six: M(0,1) at 0.11, M(0.5,0.5) at 0.17, M(1,0) at 0.45, the zero output of the non-zero input (2, −1) at 1.01, and the two negative outputs at 1.57 and 2.12. Every marker lies on the line y = x/2. A metric control switches to cosine, where the three positive outputs tie at 1.00, the two negatives score −1.00 and the zero output scores 0.00, and to the dot product, which ranks by length.

  1. Read the shape as (out features, in features) for a weight matrix, so the multiply consumes the input's last dimension.
  2. Check the inner dimensions match, and predict the output shape before running the code.
  3. Compute rank when a layer seems unable to fit: a rank-deficient weight map ignores part of its input space.
  4. Use transposes to align dimensions, remembering that a transpose changes strides rather than moving data.
  5. Batch the multiply by stacking examples in a leading dimension, so one call replaces a Python loop.
  6. Express multi-term contractions with einsum when the index bookkeeping is unclear, then check the result against explicit matmuls.

Trade-offs

ChoiceGood forCostFails when
Dense matrixFull-rank expressiveness, hardware-friendlyParameters scale with both dimensionsThe layer is huge and mostly redundant
Low-rank factorisationFar fewer parameters, cheaper adaptationRestricted to a subspace of the full mapThe task needs a full-rank transformation
Batch matmulHigh device utilisationMemory for intermediate productsBatch or head dimensions are very large
Sparse matrixMemory proportional to non-zerosIrregular access, weaker hardware supportThe matrix is mostly dense in practice
Trade-off mapInteractive

Good for

Full-rank expressiveness, hardware-friendly

Cost

Parameters scale with both dimensions

Fails when

The layer is huge and mostly redundant

  1. The layer is huge and mostly redundant
  2. The task needs a full-rank transformation
  3. Batch or head dimensions are very large
  4. The matrix is mostly dense in practice

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Dense matrix is good for Full-rank expressiveness, hardware-friendly, costs Parameters scale with both dimensions, and fails when The layer is huge and mostly redundant. Low-rank factorisation is good for Far fewer parameters, cheaper adaptation, costs Restricted to a subspace of the full map, and fails when The task needs a full-rank transformation. Batch matmul is good for High device utilisation, costs Memory for intermediate products, and fails when Batch or head dimensions are very large. Sparse matrix is good for Memory proportional to non-zeros, costs Irregular access, weaker hardware support, and fails when The matrix is mostly dense in practice

Failure modes

  • Silent broadcasting in matmul. A missing dimension is treated as a batch of one and the result has an unexpected shape. Debug: print both operand shapes and the output shape, then compare against a hand-computed expectation.
  • Rank collapse. A product of matrices loses rank and the layer cannot distinguish inputs it should. Debug: compute the singular values of the weight and check how many are effectively zero.
  • Transpose mistaken for reshape. The values land in the wrong positions while the shape looks correct. Debug: multiply by a one-hot vector and check which output coordinate becomes non-zero.

In production

Assert weight shapes against the config at load time, because a checkpoint with a transposed matrix can still load and produce plausible but degraded output. When adapting a model with low-rank updates, log the effective rank so a collapsed adapter is visible.

Common misconceptions

People think: A matrix is just a 2D array.

Actually: The array is the storage; the mathematical object is a linear map, and properties like rank and null space describe what it can and cannot represent.

People think: More parameters in a matrix always mean more capacity.

Actually: Capacity depends on rank and on training signal; a large matrix trained on little data can behave like a much smaller one.

Interview angle

A strong answer treats a weight matrix as a linear map, reads shapes fluently, and connects rank to representational limits. It should distinguish transpose from reshape and explain batching. Weak answers can multiply matrices on paper but cannot say what a rank-deficient weight implies or why shape errors dominate implementation bugs.

Sources