Why it matters
A training bill is the product of time and hardware, and both are downstream of memory. If the model plus optimiser state does not fit, you shard, and sharding adds communication. If activations do not fit, you reduce the batch size, and small batches waste device time. Estimating cost means accounting for each memory term and knowing which intervention moves which one.
Mental model
Training cost is a delivery van: the price is set by how many trips you make, and the trips are set by how much fits in the van. The analogy breaks because you can trade cargo space for fuel, which is what activation checkpointing does.
How it works
- 1Compute the memory terms per device: parameters, gradients, optimiser state and activations at your sequence length.
- 2Choose the smallest intervention that fits: smaller batch, gradient accumulation, activation checkpointing, then sharding.
- 3Estimate step time from the slowest stage, since data loading and communication often dominate over arithmetic.
- 4Run a short calibration job and extrapolate from measured steps per second rather than from a theoretical peak.
- 5Price the whole campaign: sweeps, failed runs, evaluation jobs and idle time.
- 6Record actual spend against the estimate for the next run, because the ratio is usually stable per team.
Step 1 of 6
Compute the memory terms per device: parameters, gradients, optimiser state and activations at your sequence length.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Compute the memory terms per device: parameters, gradients, optimiser state and activatio… 2. Choose the smallest intervention that fits: smaller batch, gradient accumulation, activat… 3. Estimate step time from the slowest stage, since data loading and communication often dom… 4. Run a short calibration job and extrapolate from measured steps per second rather than fr… 5. Price the whole campaign: sweeps, failed runs, evaluation jobs and idle time. 6. Record actual spend against the estimate for the next run, because the ratio is usually s…
- Compute the memory terms per device: parameters, gradients, optimiser state and activations at your sequence length.
- Choose the smallest intervention that fits: smaller batch, gradient accumulation, activation checkpointing, then sharding.
- Estimate step time from the slowest stage, since data loading and communication often dominate over arithmetic.
- Run a short calibration job and extrapolate from measured steps per second rather than from a theoretical peak.
- Price the whole campaign: sweeps, failed runs, evaluation jobs and idle time.
- Record actual spend against the estimate for the next run, because the ratio is usually stable per team.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Single device | No communication overhead | Memory ceiling | The model and optimiser state do not fit |
| Data parallel | Simple scaling | Replicated optimiser state | Memory per device is the limit |
| Sharded optimiser state | Fits larger models | Communication and complexity | The model already fits |
| Activation checkpointing | Large memory savings | Extra compute per step | Steps are already compute-bound |
Good for
No communication overhead
Cost
Memory ceiling
Fails when
The model and optimiser state do not fit
- The model and optimiser state do not fit
- Memory per device is the limit
- The model already fits
- Steps are already compute-bound
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Single device is good for No communication overhead, costs Memory ceiling, and fails when The model and optimiser state do not fit. Data parallel is good for Simple scaling, costs Replicated optimiser state, and fails when Memory per device is the limit. Sharded optimiser state is good for Fits larger models, costs Communication and complexity, and fails when The model already fits. Activation checkpointing is good for Large memory savings, costs Extra compute per step, and fails when Steps are already compute-bound
Failure modes
- Estimating from parameters only. The job runs out of memory at the first step because activations dominate. Debug: log peak allocated memory per step and compare it with the estimate for each term.
- Batch size too small. Throughput is a fraction of the device's capability and cost per example is high. Debug: sweep batch size and plot examples per second until it plateaus.
- Idle accelerator time. Cost is dominated by waiting for data or checkpoints. Debug: measure data-loader wait and checkpoint write time as a share of step time.
In production
Track cost per training run and per accepted model, and cap sweep budgets before launch. Re-measure throughput after any change to sequence length, since activation memory scales with it.
Common misconceptions
People think: Cost is proportional to parameter count.
Actually: Memory layout, sequence length and parallelism decide the bill as much as model size does.
People think: Adapters make training cheap.
Actually: They cut optimiser state and checkpoint size; activations still scale with batch and sequence length.
Interview angle
Do the memory arithmetic out loud and connect each term to an intervention. Say why a memory saving that does not raise batch size may not reduce cost. Weak answers quote a price per device hour without connecting it to throughput or to what fits.
Sources
- Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — partitioning optimiser state, gradients and parameters.
- Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models — the activation memory term and checkpointing trade-offs.
- Hoffmann et al., Training Compute-Optimal Large Language Models — how data and parameter budgets interact with compute.