Why it matters
A training run is a long-lived process whose correctness depends on details outside the model: how batches are formed, when the scheduler advances, what is logged, and what a restart restores. Most silent training failures — a shuffled validation set, a schedule advanced per micro-batch, an evaluation that reuses training data — are loop bugs rather than model bugs. Making the loop explicit is what makes runs comparable.
Mental model
The training loop is a shift schedule for a factory: each step feeds a batch, records the error, adjusts the tooling, and periodically inspects output quality. The metaphor breaks because the loop also changes its own rules as it runs — the learning rate decays, data order is reshuffled, and early stopping can end the shift before the schedule finishes.
How it works
- 1Shuffle the training split with a seeded sampler and form batches of a fixed token or example budget rather than a fixed count.
- 2For each step, zero gradients, run the forward pass, compute the loss, and call backward.
- 3Scale the loss by the accumulation factor when micro-batching so the accumulated gradient matches the intended batch.
- 4Clip the gradient norm, then let the optimiser update parameters and advance the scheduler exactly once per optimiser step.
- 5Every fixed interval, evaluate on the validation split with the model in eval mode and the sampler disabled.
- 6Checkpoint weights, optimiser state, scheduler position, sampler epoch and the random number generator state together.
Step 1 of 6
Shuffle the training split with a seeded sampler and form batches of a fixed token or example budget rather than a fixed count.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Shuffle the training split with a seeded sampler and form batches of a fixed token or exa… 2. For each step, zero gradients, run the forward pass, compute the loss, and call backward. 3. Scale the loss by the accumulation factor when micro-batching so the accumulated gradient… 4. Clip the gradient norm, then let the optimiser update parameters and advance the schedule… 5. Every fixed interval, evaluate on the validation split with the model in eval mode and th… 6. Checkpoint weights, optimiser state, scheduler position, sampler epoch and the random num…
- Shuffle the training split with a seeded sampler and form batches of a fixed token or example budget rather than a fixed count.
- For each step, zero gradients, run the forward pass, compute the loss, and call backward.
- Scale the loss by the accumulation factor when micro-batching so the accumulated gradient matches the intended batch.
- Clip the gradient norm, then let the optimiser update parameters and advance the scheduler exactly once per optimiser step.
- Every fixed interval, evaluate on the validation split with the model in eval mode and the sampler disabled.
- Checkpoint weights, optimiser state, scheduler position, sampler epoch and the random number generator state together.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Fixed example batches | Stable step cost, simple throughput planning | Variable padding and wasted compute on short examples | Sequence lengths vary widely |
| Fixed token batches | Even compute per step, better packing | More complex batching and loss weighting | The implementation cannot mask padding correctly |
| Evaluate every epoch | Cheap, low overhead | Late detection of divergence | Runs are long or data is expensive |
| Evaluate every N steps | Early signal, comparable across runs | Evaluation cost and possible overfitting to the validation set | The validation split is small |
Good for
Stable step cost, simple throughput planning
Cost
Variable padding and wasted compute on short examples
Fails when
Sequence lengths vary widely
- Sequence lengths vary widely
- The implementation cannot mask padding correctly
- Runs are long or data is expensive
- The validation split is small
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Fixed example batches is good for Stable step cost, simple throughput planning, costs Variable padding and wasted compute on short examples, and fails when Sequence lengths vary widely. Fixed token batches is good for Even compute per step, better packing, costs More complex batching and loss weighting, and fails when The implementation cannot mask padding correctly. Evaluate every epoch is good for Cheap, low overhead, costs Late detection of divergence, and fails when Runs are long or data is expensive. Evaluate every N steps is good for Early signal, comparable across runs, costs Evaluation cost and possible overfitting to the validation set, and fails when The validation split is small
Failure modes
- Scheduler advanced per micro-batch. The learning rate decays faster than intended and the run stalls early. Debug: log the learning rate beside the optimiser step count and check the ratio against the schedule definition.
- Validation split not held fixed. Metrics improve because evaluation examples change between checks. Debug: hash the evaluation ids once at startup and assert the hash on every evaluation.
- Restart loses data position. Resumed runs repeat or skip examples and the loss curve has a discontinuity. Debug: compare the loss on the step before and after a forced restart with identical seeds.
In production
Emit step time, tokens per second, gradient norm and learning rate on every step, and loss on a fixed cadence. Rehearse a restart from checkpoint in staging, because an untested resume path is the most common cause of lost training days.
Common misconceptions
People think: An epoch and a step are both just iterations.
Actually: A step is one optimiser update over one batch; an epoch is a full pass. With gradient accumulation and variable batch sizes the two diverge, and schedules written against the wrong unit misbehave.
People think: Checkpointing the weights is enough to resume.
Actually: Without optimiser moments, scheduler position and data order, a resumed run follows a different trajectory and is not comparable to an uninterrupted one.
Interview angle
A strong answer describes the loop as a state machine: data order, forward, backward, accumulation, clipping, step, schedule, evaluation, checkpoint. It should name what must be saved to resume and which bugs are loop bugs rather than model bugs. Weak answers describe 'train for some epochs' and cannot say how accumulation interacts with the scheduler.
Sources
- PyTorch, Training a model tutorial — the canonical loop structure and eval-mode handling.
- Goodfellow, Bengio & Courville, Deep Learning — batch training, epochs and stochastic optimisation.
- PyTorch, Reproducibility — seeding, determinism and what a restart must restore.