Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 6 of 33

Tensors

0 of 33 complete

Why it matters

Model code is mostly tensor plumbing: batch dimensions, head splits, transposes before a matmul, masks that must broadcast over the right axis. Frameworks make these operations cheap by returning views into shared memory, which means a correct-looking expression can produce an incorrect gradient or a silent aliasing bug. Understanding shape, stride and contiguity is what turns a shape error from a mystery into a one-line fix.

Mental model

A tensor is a window onto a flat warehouse shelf: shape says how you are allowed to index, strides say how many shelf positions each index step jumps. The metaphor breaks because several windows can overlap the same shelf, so two tensors that look independent may be the same bytes.

How it works

Strides · one buffer, three address mapsInteractive

hover or focus a cell

015
logical row ↓logical col →
c0c1c2c3
r0
r1
r2
r3

Hover or focus a cell to read its exact value. Shade and printed number encode the same quantity.

shape
4 × 4
strides
(4, 1)
contiguous
yes
bytes at fp32
16 × 4 = 64

Step 1 of 3

Row-major · (4, 1)

With a row step of 4 and a column step of 1, reading left to right walks 0 to 15 with no gaps. This is the layout a matmul kernel prefers, and it is why the offset of (r, c) is 4r + c.

Next: Transposed view · (1, 4)

Step 1/3 · Row-major · (4, 1)

Figure. A 4 × 4 logical grid over one 16-element buffer, illustrative strides. Each cell prints the element's offset, not its value, so the grid shows how a (row, column) index maps to memory. Row-major strides (4, 1) give offsets 0–15 in reading order; the transposed view of the same square tensor has strides (1, 4) and gives the same sixteen offsets in a different order; a row stride of 0 repeats the same four offsets without copying. The printed offsets are exact for the strides shown.

What to notice. The transposed view's row step is one element and its column step is four, so reading across a logical row walks 0, 4, 8 and 12: four elements apart, not contiguous. That is why a kernel may copy before a matmul even though the shape is unchanged. The stride-0 frame is broadcasting: four logical rows share one row of four values, so sixteen cells cost four stored elements and no copy. This is a float32 buffer of 16 × 4 = 64 bytes with no storage offset and no negative strides; a slice can start partway in and carry an offset the grid does not show.

Text version

Three frames of a four by four logical grid; each cell prints a memory offset. The first frame has row-major strides four and one and prints 0, 1, 2, 3; 4, 5, 6, 7; 8, 9, 10, 11; 12, 13, 14, 15. The second has transposed strides one and four and prints 0, 4, 8, 12; 1, 5, 9, 13; 2, 6, 10, 14; 3, 7, 11, 15. The third has a row stride of zero and prints 0, 1, 2, 3 in all four rows. Steps highlight the row-major order, the transposed row and the broadcast rows; the readout reports the logical row, the logical column and the printed offset.

  1. Fix a dimension convention for the codebase. for example batch, sequence, channel — and assert it at every function boundary.
  2. Inspect shape, dtype and device together before debugging values, because a cast or a move changes results as much as a reshape.
  3. Prefer view-style operations for speed, but call contiguous before an operation that requires a packed layout.
  4. Make broadcasting explicit with unsqueeze at the intended axis rather than relying on implicit alignment from the right.
  5. Use named or commented dimensions for attention code, where head, query and key axes are easy to swap silently.
  6. Check that gradients flow through view operations by comparing a small autograd result against a manual computation.

Trade-offs

ChoiceGood forCostFails when
Views over copiesCheap, no extra memoryAliasing bugs and unexpected gradient writesYou mutate a tensor another branch still needs
Contiguous copiesPredictable layout for kernelsMemory traffic and allocationTensors are large and copied per step
High-rank explicit tensorsClear per-head and per-batch indexingMore reshape code and more axes to get wrongThe code is written once and never inspected
Implicit broadcastingTerse, readable arithmeticSilent expansion on the wrong axisTwo dimensions of size one are both present
Trade-off mapInteractive

Good for

Cheap, no extra memory

Cost

Aliasing bugs and unexpected gradient writes

Fails when

You mutate a tensor another branch still needs

  1. You mutate a tensor another branch still needs
  2. Tensors are large and copied per step
  3. The code is written once and never inspected
  4. Two dimensions of size one are both present

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Views over copies is good for Cheap, no extra memory, costs Aliasing bugs and unexpected gradient writes, and fails when You mutate a tensor another branch still needs. Contiguous copies is good for Predictable layout for kernels, costs Memory traffic and allocation, and fails when Tensors are large and copied per step. High-rank explicit tensors is good for Clear per-head and per-batch indexing, costs More reshape code and more axes to get wrong, and fails when The code is written once and never inspected. Implicit broadcasting is good for Terse, readable arithmetic, costs Silent expansion on the wrong axis, and fails when Two dimensions of size one are both present

Failure modes

  • Transpose without contiguous. A matmul receives a non-contiguous view and a kernel either copies implicitly or fails on an unsupported layout. Debug: check is_contiguous and the stride tuple at the failing call.
  • Broadcast on the wrong axis. A mask of shape (batch, 1) is meant to align with sequence but expands over heads instead. Debug: print the shape after the operation and compare against an explicit unsqueeze version.
  • Aliased in-place update. Two tensors share storage, so normalising one corrupts the other. Debug: compare data_ptr of the inputs and outputs of the suspect operation.

In production

Assert shapes and dtypes at module boundaries in debug builds, and keep those assertions in the hot path only when they are cheap. When exporting a model, verify that the traced graph preserves the same layout assumptions, since export can bake in contiguity that the original code left implicit.

Common misconceptions

People think: Reshape and transpose are interchangeable.

Actually: Reshape changes the logical shape without moving data; transpose changes the stride mapping. Confusing them produces tensors with the right shape and the wrong element order.

People think: A tensor's shape tells you its memory layout.

Actually: Shape describes indexing, not storage; two tensors with identical shapes can have different strides and different performance characteristics.

Interview angle

A strong answer distinguishes shape from stride, explains views versus copies, and gives a concrete failure such as a mask broadcasting on the head axis. It should connect layout to kernel performance. Weak answers define a tensor as a multidimensional array and stop, which misses the aliasing and contiguity issues that dominate real debugging.

Sources