Why it matters
Two models with the same parameter count can run at very different speeds because the hardware has a roofline: below a threshold of work per byte, adding compute does nothing. Practitioners who plan from peak FLOPs buy the wrong hardware or mis-size batches. Understanding the memory hierarchy explains why flash attention, fused kernels and quantisation help, and why a bigger GPU does not fix a bandwidth-bound decode loop.
Mental model
An accelerator is a kitchen with a huge stove and a tiny counter: cooking is fast, but every ingredient must be carried from the pantry. The metaphor breaks because the pantry is not slow in bandwidth terms — it is enormous and fast, just far away, so the cost is the trip and the counter space, not the fetching speed.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.
- tile
- 4 × 4 = 16 cells
- K
- 4
- MACs per tile
- 16 × 4 = 64
- live cells
- 16/16 = 100%
Step 1 of 4
Full tile · K = 4
Every output cell is present and every reduction step has work. The row totals are 16 MACs per output row and the column totals are 16 per output column; 64 is the whole tile's arithmetic.
Next: Ragged batch · 3 rows
Figure. One 4 × 4 output tile on a 4 × 4 multiply-accumulate array, illustrative schedule. Each cell prints the multiply-accumulates it performs in one tile pass, and K is the reduction length. A full tile keeps all 16 cells busy; a ragged batch row or a single decode row leaves whole rows idle. Row and column totals add the printed MACs, so the live share is visible as arithmetic rather than inferred from shade. The counts are a schedule, not a measurement.
What to notice. Utilisation is the live-cell share, not the arithmetic total. Twelve of sixteen cells is 75% at K = 4, and four of sixteen is 25% for a single decode row; the idle cells are printed as 0 rather than left blank, because the row exists and does no work. The last frame raises K to 16: every cell stays live and each cell does 16 MACs, so the tile performs 16 × 16 = 256 and the array is still fully occupied. Longer K does not fix a ragged batch; only shape does. A real kernel tiles, splits K and schedules several tiles per array, so one 4 × 4 grid is an analogy for occupancy, not a cycle-accurate trace.
Text version
Four frames of a four by four tile. Rows and columns are the sixteen output cells of one tile. In the first frame every cell prints 4, the row totals read 16 and the column totals read 16, for 64 multiply-accumulates across the tile. In the second frame the fourth row prints 0 and the column totals fall to 12, for 48 MACs and 12 of 16 live cells. In the third frame only the first row prints 4, the column totals read 4, and 4 of 16 cells are live. In the fourth frame every cell prints 16, the row totals read 64 and the column totals read 64, for 256 MACs. Steps highlight the ragged fourth row, the three idle decode rows and the full K = 16 tile; the readout reports the output row, the output column and the printed MAC count.
- Estimate arithmetic intensity for the operation: FLOPs divided by bytes read and written, including intermediate tensors.
- Compare it against the device's compute-to-bandwidth ratio to see which side of the roofline you are on.
- Tile the computation so reused operands stay in registers or shared memory instead of returning to high-bandwidth memory.
- Fuse elementwise operations into the producing kernel to avoid writing and re-reading full tensors.
- Choose the dtype and accumulate dtype deliberately; tensor cores need specific shapes and precisions to engage.
- For multi-device runs, measure the communication volume per step and overlap it with computation where the topology allows.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Large fused kernels | Less memory traffic, higher utilisation | Hard to write, backend-specific, harder to debug | The shape or dtype falls outside the supported path |
| Higher precision throughout | Numerical safety, simpler reasoning | Half the throughput and double the memory | Throughput or context length is the constraint |
| Data parallel across devices | Simple scaling for models that fit | Gradient all-reduce every step | Interconnect is slow relative to compute |
| Model parallel | Trains models too large for one device | Frequent activation and gradient transfers | Layers are small relative to communication cost |
Good for
Less memory traffic, higher utilisation
Cost
Hard to write, backend-specific, harder to debug
Fails when
The shape or dtype falls outside the supported path
- The shape or dtype falls outside the supported path
- Throughput or context length is the constraint
- Interconnect is slow relative to compute
- Layers are small relative to communication cost
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Large fused kernels is good for Less memory traffic, higher utilisation, costs Hard to write, backend-specific, harder to debug, and fails when The shape or dtype falls outside the supported path. Higher precision throughout is good for Numerical safety, simpler reasoning, costs Half the throughput and double the memory, and fails when Throughput or context length is the constraint. Data parallel across devices is good for Simple scaling for models that fit, costs Gradient all-reduce every step, and fails when Interconnect is slow relative to compute. Model parallel is good for Trains models too large for one device, costs Frequent activation and gradient transfers, and fails when Layers are small relative to communication cost
Failure modes
- Bandwidth-bound decode. Adding compute or a bigger GPU barely changes tokens per second because the loop is reading weights. Debug: compare achieved bandwidth against the device's peak; a value near peak means memory is the limit.
- Tensor cores idle. Matmuls run on the slower general path because dimensions or dtypes do not match the required shapes. Debug: profile one step and check the kernel names and achieved FLOPs against the theoretical peak.
- Communication-dominated scaling. More devices reduce per-device work but total step time rises. Debug: measure time in the all-reduce separately from the compute phase for the same step.
In production
Track achieved memory bandwidth and achieved FLOPs per kernel, not just device utilisation, which can read high while the device waits. Rehearse the failure mode where one device straggles, since synchronous training makes the slowest rank set step time.
Common misconceptions
People think: A higher peak-FLOP device always trains faster.
Actually: Peak FLOPs only matter for compute-bound kernels at high arithmetic intensity; memory-bound work improves with bandwidth and cache, not with more arithmetic units.
People think: GPU utilisation near 100 percent means the job is efficient.
Actually: Utilisation counts busy time, and a device can be busy waiting on memory. The useful measures are achieved bandwidth, achieved FLOPs and step time against the roofline.
Interview angle
A strong answer starts from the roofline: arithmetic intensity, the compute-to-bandwidth ratio, and which regime the workload sits in. It should explain tiling, fusion and dtype choice as consequences, and separate prefill from decode. Weak answers quote device specifications without connecting them to the operation being run.
Sources
- Williams, Waterman & Patterson, Roofline: An Insightful Visual Performance Model — the compute-versus-bandwidth bound framing.
- NVIDIA, CUDA C Programming Guide — the memory hierarchy, tiling and execution model.
- Horace He, Making Deep Learning Go Brrrr From First Principles — a practical account of compute, memory and overhead bottlenecks.