Why it matters
System design interviews and real architecture reviews both need a sanity check before implementation. Will this context fit, how much GPU memory does concurrency require, how many requests per second can one worker sustain, what does a day of traffic cost in tokens. Exact answers need measurement, but order-of-magnitude answers are enough to choose between designs. The skill is knowing which quantities multiply and which assumptions dominate.
Mental model
It is a Fermi estimate for a kitchen: covers, dishes per hour, oven capacity. The metaphor breaks because LLM capacity is not additive, since batching and cache reuse make throughput depend on the mix of requests, so two designs with the same average load can need different hardware.
How it works
- 1Write the quantities you know and the ones you are guessing, and mark the guesses explicitly.
- 2Estimate tokens from characters or words, then check against a tokenizer for one real sample.
- 3Compute context cost per request as input plus output tokens, and multiply by requests per day.
- 4Size KV cache from layers, key and value heads, head dimension, sequence length, batch size and bytes per element.
- 5Convert to time using a throughput assumption, and state whether it is prefill-bound or decode-bound.
- 6Sanity-check the result against a known system or a quick measurement, and report the range rather than a single figure.
Step 1 of 6
Write the quantities you know and the ones you are guessing, and mark the guesses explicitly.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Write the quantities you know and the ones you are guessing, and mark the guesses explici… 2. Estimate tokens from characters or words, then check against a tokenizer for one real sam… 3. Compute context cost per request as input plus output tokens, and multiply by requests pe… 4. Size KV cache from layers, key and value heads, head dimension, sequence length, batch si… 5. Convert to time using a throughput assumption, and state whether it is prefill-bound or d… 6. Sanity-check the result against a known system or a quick measurement, and report the ran…
- Write the quantities you know and the ones you are guessing, and mark the guesses explicitly.
- Estimate tokens from characters or words, then check against a tokenizer for one real sample.
- Compute context cost per request as input plus output tokens, and multiply by requests per day.
- Size KV cache from layers, key and value heads, head dimension, sequence length, batch size and bytes per element.
- Convert to time using a throughput assumption, and state whether it is prefill-bound or decode-bound.
- Sanity-check the result against a known system or a quick measurement, and report the range rather than a single figure.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Round aggressively | Fast and robust to wrong inputs | Can hide a factor-of-two problem | The decision sits near a capacity limit |
| Measure one sample | Anchors the estimate | Only valid for that input shape | Traffic has a long tail of long inputs |
| Assume full cache reuse | Optimistic throughput | Overcommits memory | Every request has a unique prefix |
| Assume no batching | Conservative latency | Overestimates hardware needs | The serving stack batches by default |
Good for
Fast and robust to wrong inputs
Cost
Can hide a factor-of-two problem
Fails when
The decision sits near a capacity limit
- The decision sits near a capacity limit
- Traffic has a long tail of long inputs
- Every request has a unique prefix
- The serving stack batches by default
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Round aggressively is good for Fast and robust to wrong inputs, costs Can hide a factor-of-two problem, and fails when The decision sits near a capacity limit. Measure one sample is good for Anchors the estimate, costs Only valid for that input shape, and fails when Traffic has a long tail of long inputs. Assume full cache reuse is good for Optimistic throughput, costs Overcommits memory, and fails when Every request has a unique prefix. Assume no batching is good for Conservative latency, costs Overestimates hardware needs, and fails when The serving stack batches by default
Failure modes
- Unit slip. A per-token figure is compared with a per-request figure and the conclusion is off by the context length. Debug: write units beside every number and cancel them explicitly.
- Ignoring output length. Cost and latency are estimated from input alone and generation dominates. Debug: compute input and output contributions separately.
- Cache double count. KV cache is sized per request and then multiplied by concurrency twice. Debug: recompute memory with concurrency set to one and check the scaling is linear.
In production
Keep a small sheet of measured constants for your stack, such as tokens per character and achieved tokens per second, and replace guesses with them after the first load test.
Common misconceptions
People think: You need exact numbers to make a design decision.
Actually: Order-of-magnitude estimates separate viable designs from impossible ones, which is usually the decision at hand.
People think: Tokens map cleanly to words.
Actually: The ratio varies by language and content, so a rough factor is fine for estimation and wrong for billing.
Interview angle
Strong answers state assumptions, compute in a clear order, keep units visible and give a range with the dominant uncertainty. They notice when the result is close to a limit. Weak answers produce a confident number with no derivation and cannot say what would change it.
Sources
- Vaswani et al., Attention Is All You Need — the architecture whose dimensions drive the estimates.
- Kwon et al., PagedAttention and vLLM — KV cache memory and why concurrency is memory-bound.
- Dean & Barroso, The Tail at Scale — why capacity estimates must account for the tail.