Why it matters
Every training step is a forward pass followed by a backward pass, and every production request is a forward pass alone. The forward pass is where numerical problems become visible: a badly scaled input produces huge logits, a wrong mask leaks information, a dtype mismatch loses precision. It also decides training memory, because autograd must keep the intermediate values that the chain rule will need.
Mental model
The forward pass is a factory run with the machines frozen at their current settings: material enters, each station transforms it, and a finished product comes out. The backward pass then walks the line in reverse asking each station how it contributed. The metaphor breaks because the line keeps every intermediate part on the floor, and that storage, not the machines, is what limits batch size.
How it works
hover or focus a cell
Hover or focus a cell to read its exact value and printed row total. Shade and printed number encode the same quantity.
- input x
- 1.5 · −0.5 · 0.8
- weights, row 1
- 0.80 · −0.60 · 0.50
- pre-activation z
- 1.90 · −1.00 · 0.35
Step 1 of 4
Three outputs at once
The layer's whole forward arithmetic is nine products in one grid. Every row is one output unit's sum; every column is one input unit's fan-out.
Next: Row y₁ · a sum of three
Figure. One linear layer with three inputs and three outputs, illustrative values. Cell (i, j) is wᵢⱼ · xⱼ, the contribution of input j to output i, with x = 1.5, −0.5 and 0.8; the row total is the pre-activation zᵢ before the bias. Shade is one signed scale from −1.20 to +1.20, and every cell prints its signed product.
What to notice. The arithmetic is visible end to end: 0.80 × 1.5 = 1.20, −0.60 × −0.5 = 0.30 and 0.50 × 0.8 = 0.40 add to z₁ = 1.90. Read across a row to see one output sum its three inputs; read down a column to see one input fan out to all three outputs, with the sign of the weight flipping which way it pushes. The pass stores the input row and the output z, not these nine products, and that saved state is what the backward pass consumes and what sets activation memory.
Text version
A three by three matrix. Rows are output units y₁ to y₃ and columns input units x₁ to x₃, with x equal to 1.5, −0.5 and 0.8. Each cell prints a product of a weight and an input. Row y₁ prints 1.20, 0.30 and 0.40 and totals z₁ = 1.90; row y₂ prints −0.60, −0.60 and 0.20 and totals −1.00; row y₃ prints 0.45, 0.30 and −0.40 and totals 0.35. Shade runs teal below zero to accent above, on one scale from −1.20 to +1.20. Steps highlight the first row, the second column and the second row; the readout reports the output unit, the input unit and the printed product with its row total.
- Move the batch to the compute device and cast it to the model dtype, checking that labels stay in the dtype the loss expects.
- Embed or project the input into the residual stream, adding positional information where the architecture requires it.
- Run each block in order, storing the tensors that backward will need: normalised inputs, attention weights or their inputs, and activation outputs.
- Apply the final normalisation and the output head to produce logits over the vocabulary or classes.
- Compute the loss outside the model so the training loop owns the reduction and the logging.
- Wrap the whole call in no-grad mode for inference so no activation graph is retained.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Store all activations | Fastest backward, simplest code | Memory grows with batch, sequence and depth | Batch size is memory-bound |
| Gradient checkpointing | Much lower memory, larger batches | A second forward pass during backward | Step time is the constraint |
| Lower-precision forward | Faster matmuls, less memory traffic | Numerical drift in reductions and softmax | The task is sensitive to small logit changes |
| Fused forward kernels | Less memory traffic, higher utilisation | Harder to debug, backend-specific | You need exact parity with a reference implementation |
Good for
Fastest backward, simplest code
Cost
Memory grows with batch, sequence and depth
Fails when
Batch size is memory-bound
- Batch size is memory-bound
- Step time is the constraint
- The task is sensitive to small logit changes
- You need exact parity with a reference implementation
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Store all activations is good for Fastest backward, simplest code, costs Memory grows with batch, sequence and depth, and fails when Batch size is memory-bound. Gradient checkpointing is good for Much lower memory, larger batches, costs A second forward pass during backward, and fails when Step time is the constraint. Lower-precision forward is good for Faster matmuls, less memory traffic, costs Numerical drift in reductions and softmax, and fails when The task is sensitive to small logit changes. Fused forward kernels is good for Less memory traffic, higher utilisation, costs Harder to debug, backend-specific, and fails when You need exact parity with a reference implementation
Failure modes
- Exploding logits. Inputs or weights are mis-scaled and the forward output saturates the loss. Debug: print the maximum absolute logit and the input norm for the first batch; both should be finite and modest.
- Mask applied too late. A causal or padding mask is added after the softmax, so the forward pass already mixed forbidden positions. Debug: change a future token and confirm earlier logits are unchanged.
- Dtype mismatch at the head. Logits come out in half precision while the loss expects float32, and the loss silently loses resolution. Debug: assert dtype at each module boundary for one batch.
In production
Record logit statistics and activation norms for a fixed canary batch on every deploy; the forward pass is the cheapest place to catch a numerical regression. Cap batch size from measured activation memory rather than from the largest batch that happened to fit once.
Common misconceptions
People think: The forward pass is the cheap half of training.
Actually: It is cheaper in FLOPs than backward, but it dominates memory because every intermediate must be retained for the gradient.
People think: Inference and training forward passes are the same code path.
Actually: Inference drops gradient tracking, uses a key-value cache and often different kernels, so parity between them has to be tested rather than assumed.
Interview angle
A strong answer walks the tensor from input to logits, names what is stored for backward, and explains why activation memory rather than parameter memory sets batch size. It should distinguish the training and inference paths. Weak answers describe the pass as a black box and cannot say what makes it memory-heavy or where a mask bug would show up.
Sources
- PyTorch documentation — tensor semantics, modules and no-grad evaluation.
- PyTorch, Autograd mechanics — which intermediates are saved and how graph retention works.
- PyTorch, torch.no_grad — disabling graph construction for inference.