Why it matters
A model that fits during a single-request test can fail under production concurrency, because each additional sequence needs its own key and value tensors for every layer and token. Teams size hardware from parameter counts alone, then discover that the cache, the activation workspace and the allocator's reserved blocks were never in the budget. Planning exists to make those terms explicit before a deployment fails at three in the morning.
Mental model
Memory is a workshop bench: weights are the tools bolted to it, the KV cache is the material laid out for each job in progress, and activations are the shavings. The analogy breaks because the bench cannot be cleared mid-job, and the allocator keeps reserved space that looks empty but cannot be handed to another request.
How it works
Memory budget · GiB
11.5 of 24 GiB used
Budget lines · GiB
- Weights8 GiB
- Runtime + buffers2 GiB
- Activations0.5 GiB
- KV cache0 GiB
- Headroom1 GiB
Step · 1 of 8
Weights loaded
The 8B model at 8 GiB (FP8, rounded) plus 2 GiB of runtime and buffers, 0.5 GiB of activation workspace and 1 GiB of deliberate headroom total 11.5 GiB before any request. The card has 12.5 GiB free, of which 1 GiB is reserved headroom, so 11.5 GiB is usable by cache.
Next: Two sequences in flight
- KV cache (GiB)
- 0
- Sequences
- 0
- Free (GiB)
- 12.5
- Rejected
- 0
Figure. A 24 GiB card holding an 8B model at 8 GiB (FP8, rounded), 2 GiB runtime, 1.5 GiB activations, 1 GiB headroom and 1 GiB of KV cache per 8,192-token sequence (128 KiB/token, the Phase A arithmetic). The cache is drawn in 1 GiB sequence blocks as a zoom; real engines allocate smaller fixed blocks. Single process only: fragmentation and a co-tenant are omitted. Synthetic budget, not a measured deployment.
What to notice. Weights do not move, so they are not what fails at peak. The cache term is the only line that grows with traffic: at eleven sequences the card has 0.5 GiB left and the twelfth request cannot be placed. Step 6 draws that demand past the marker; step 7 is the honest fix, capping concurrency so the rejection happens before allocation. Step 8 frees 4 GiB of weights and buys four more sequences, not double, because the cache per sequence is unchanged.
Text version
A stacked memory bar with a 24 GiB capacity marker. Five segments: weights 8 GiB, runtime and buffers 2 GiB, activations 0.5 to 1.5 GiB, KV cache 0 to 11 GiB, headroom 1 GiB. Step 1 loads the model: 11.5 GiB used, 12.5 free. Step 2: two sequences, 2 GiB cache, 13.5 used. Step 3: six sequences, 6 GiB cache, 18 used. Step 4: ten sequences, 10 GiB cache, 22.5 used. Step 5: eleven sequences, 11 GiB cache, 23.5 used, 0.5 free. Step 6: a twelfth sequence needs 1 GiB; the demand is drawn in red beyond the 24 GiB marker, labelled over cap, at 24.5 GiB. Step 7: concurrency is capped at eleven, cache 11 GiB, 0.5 GiB free, one request rejected before allocation. Step 8: a 4-bit checkpoint halves weights to 4 GiB and the cache grows to 15 GiB for fifteen sequences, still 0.5 GiB free; the sequence count rises from eleven to fifteen, not double. Counters show cache, sequences, free GiB and rejected admissions for every step.
- Compute weight memory as parameter count times bytes per parameter at load precision, then add adapters and embedding tables separately.
- Compute cache bytes per token as two tensors times layers times key-value heads times head dimension times cache precision.
- Multiply by context length and intended concurrency; this product is the number that usually exceeds the budget.
- Add activation workspace, device context, communication buffers and a deliberate headroom fraction.
- Check whether the engine reserves the cache up front or grows it, since the two fail differently under load.
- Verify with a device query during a realistic load test, and compare predicted against observed allocation.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Load in lower precision | Fits larger models, leaves cache headroom | Quality loss that is task-dependent | The workload depends on fine numerical distinctions |
| Longer maximum context | Fewer truncations on long documents | Cache reserved per sequence grows with the window | Concurrency matters more than document length |
| Higher concurrency | Better throughput per card | Each sequence holds cache for its whole life | Requests arrive with long prompts |
| Tensor parallelism | A model that does not fit on one device | Weights shrink per card but communication buffers appear | The interconnect is slow relative to compute |
Good for
Fits larger models, leaves cache headroom
Cost
Quality loss that is task-dependent
Fails when
The workload depends on fine numerical distinctions
- The workload depends on fine numerical distinctions
- Concurrency matters more than document length
- Requests arrive with long prompts
- The interconnect is slow relative to compute
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Load in lower precision is good for Fits larger models, leaves cache headroom, costs Quality loss that is task-dependent, and fails when The workload depends on fine numerical distinctions. Longer maximum context is good for Fewer truncations on long documents, costs Cache reserved per sequence grows with the window, and fails when Concurrency matters more than document length. Higher concurrency is good for Better throughput per card, costs Each sequence holds cache for its whole life, and fails when Requests arrive with long prompts. Tensor parallelism is good for A model that does not fit on one device, costs Weights shrink per card but communication buffers appear, and fails when The interconnect is slow relative to compute
Failure modes
- Cache exhaustion at peak. Requests are admitted until the allocator cannot find a block, then the engine preempts or crashes. Debug: log cache utilisation and rejected admissions against concurrency in a load test.
- Fragmentation after long requests. Free memory exists in small pieces and a new long sequence cannot be placed. Debug: compare total free bytes with the largest contiguous allocation the engine reports.
- Hidden co-tenant. Another process on the same card holds memory and the serving process sees less than capacity. Debug: list per-process device memory and check the expected total against the card.
In production
Export cache utilisation, preemption count and rejected requests, and alert on sustained utilisation rather than a single spike. Cap maximum context and concurrency explicitly so the failure is a clean rejection instead of an allocator error.
Common misconceptions
People think: If the weights fit, the model will serve.
Actually: Weights are the smallest term under concurrent traffic; the cache grows with sequences and context and usually binds first.
People think: Quantising weights doubles concurrency.
Actually: It frees a fixed amount of memory; whether that converts into concurrency depends on how much cache each extra sequence needs.
Interview angle
Write the budget as a sum: weights, cache, activations, overhead, headroom, and show which term scales with traffic. Then explain the engine's allocation strategy and what happens when the cache runs out. Quoting a parameter count and a card size without the per-token cache term is the common miss.
Sources
- EleutherAI, Transformer Math 101 — parameter, activation and cache memory arithmetic.
- Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention — why the KV cache dominates serving memory.