Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 12 of 33

Back-of-envelope math

0 of 33 complete

Why it matters

System design interviews and real architecture reviews both need a sanity check before implementation. Will this context fit, how much GPU memory does concurrency require, how many requests per second can one worker sustain, what does a day of traffic cost in tokens. Exact answers need measurement, but order-of-magnitude answers are enough to choose between designs. The skill is knowing which quantities multiply and which assumptions dominate.

Mental model

It is a Fermi estimate for a kitchen: covers, dishes per hour, oven capacity. The metaphor breaks because LLM capacity is not additive, since batching and cache reuse make throughput depend on the mix of requests, so two designs with the same average load can need different hardware.

How it works

Mechanism, step by stepInteractive
  1. 1Write the quantities you know and the ones you are guessing, and mark the guesses explicitly.
  2. 2Estimate tokens from characters or words, then check against a tokenizer for one real sample.
  3. 3Compute context cost per request as input plus output tokens, and multiply by requests per day.
  4. 4Size KV cache from layers, key and value heads, head dimension, sequence length, batch size and bytes per element.
  5. 5Convert to time using a throughput assumption, and state whether it is prefill-bound or decode-bound.
  6. 6Sanity-check the result against a known system or a quick measurement, and report the range rather than a single figure.

Step 1 of 6

Write the quantities you know and the ones you are guessing, and mark the guesses explicitly.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Write the quantities you know and the ones you are guessing, and mark the guesses explici… 2. Estimate tokens from characters or words, then check against a tokenizer for one real sam… 3. Compute context cost per request as input plus output tokens, and multiply by requests pe… 4. Size KV cache from layers, key and value heads, head dimension, sequence length, batch si… 5. Convert to time using a throughput assumption, and state whether it is prefill-bound or d… 6. Sanity-check the result against a known system or a quick measurement, and report the ran…

  1. Write the quantities you know and the ones you are guessing, and mark the guesses explicitly.
  2. Estimate tokens from characters or words, then check against a tokenizer for one real sample.
  3. Compute context cost per request as input plus output tokens, and multiply by requests per day.
  4. Size KV cache from layers, key and value heads, head dimension, sequence length, batch size and bytes per element.
  5. Convert to time using a throughput assumption, and state whether it is prefill-bound or decode-bound.
  6. Sanity-check the result against a known system or a quick measurement, and report the range rather than a single figure.

Trade-offs

ChoiceGood forCostFails when
Round aggressivelyFast and robust to wrong inputsCan hide a factor-of-two problemThe decision sits near a capacity limit
Measure one sampleAnchors the estimateOnly valid for that input shapeTraffic has a long tail of long inputs
Assume full cache reuseOptimistic throughputOvercommits memoryEvery request has a unique prefix
Assume no batchingConservative latencyOverestimates hardware needsThe serving stack batches by default
Trade-off mapInteractive

Good for

Fast and robust to wrong inputs

Cost

Can hide a factor-of-two problem

Fails when

The decision sits near a capacity limit

  1. The decision sits near a capacity limit
  2. Traffic has a long tail of long inputs
  3. Every request has a unique prefix
  4. The serving stack batches by default

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Round aggressively is good for Fast and robust to wrong inputs, costs Can hide a factor-of-two problem, and fails when The decision sits near a capacity limit. Measure one sample is good for Anchors the estimate, costs Only valid for that input shape, and fails when Traffic has a long tail of long inputs. Assume full cache reuse is good for Optimistic throughput, costs Overcommits memory, and fails when Every request has a unique prefix. Assume no batching is good for Conservative latency, costs Overestimates hardware needs, and fails when The serving stack batches by default

Failure modes

  • Unit slip. A per-token figure is compared with a per-request figure and the conclusion is off by the context length. Debug: write units beside every number and cancel them explicitly.
  • Ignoring output length. Cost and latency are estimated from input alone and generation dominates. Debug: compute input and output contributions separately.
  • Cache double count. KV cache is sized per request and then multiplied by concurrency twice. Debug: recompute memory with concurrency set to one and check the scaling is linear.

In production

Keep a small sheet of measured constants for your stack, such as tokens per character and achieved tokens per second, and replace guesses with them after the first load test.

Common misconceptions

People think: You need exact numbers to make a design decision.

Actually: Order-of-magnitude estimates separate viable designs from impossible ones, which is usually the decision at hand.

People think: Tokens map cleanly to words.

Actually: The ratio varies by language and content, so a rough factor is fine for estimation and wrong for billing.

Interview angle

Strong answers state assumptions, compute in a clear order, keep units visible and give a range with the dominant uncertainty. They notice when the result is close to a limit. Weak answers produce a confident number with no derivation and cannot say what would change it.

Sources