Why it matters
Latency complaints arrive as a single number, usually an average, and the work is to turn that into a stage budget. Under drill conditions the interviewer watches whether you measure before changing and whether you can name the trade-off in your fix. Adding a cache or a smaller model without a stage breakdown is a guess that sometimes helps and cannot be defended. The strong answer is a short diagnostic sequence with one change and a re-measurement.
Mental model
It is a relay race with a stopwatch at each handover: the total is the sum, and the slowest leg sets the floor. The metaphor breaks because legs can sometimes run in parallel, so the critical path is not always the longest single stage.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Cumulative latency distributions for one route before and after a single change: the share of requests finished at or below each latency. The log-normal models share a family but not a tail, and the cache curve crosses the baseline near 1,500 ms — faster below it, slower above. Constants are illustrative.
What to notice. The cache fix moves the median from 900 to 750 ms — a 17% win on the headline number — while p95 rises from 2,220 to 2,580 ms and p99 from 3,240 to 4,290 ms. The curves cross near 1,500 ms: every request below the crossing is faster and the slow slice is worse. Parallel retrieval moves the whole curve: median 620 ms, p95 1,370 ms.
Text version
Cumulative share of requests served against latency from 0 to 4,000 ms. The baseline runs from a 900 ms median to 2,220 ms at p95 and 3,240 ms at p99. After caching, the curve is above the baseline until about 1,500 ms and below it after: median 750 ms, p95 2,580 ms, p99 4,290 ms. After parallel retrieval the whole curve shifts left: median 620 ms and p95 1,370 ms. A three-button control selects no change, caching or parallel retrieval, a vertical line marks an example 2,000 ms budget, and a panel reports p50, p95 and p99 for the selected frame.
- Ask for p50 and p95, the route, and whether the complaint is about first token or full response.
- Request a trace with one span per stage, including retrieval, reranking, queue wait and generation.
- Compute the share of total time per stage and identify the critical path.
- Check for serial calls that could run concurrently and for retries that duplicate a stage.
- Propose one change with its expected effect and its quality or cost.
- Re-measure the same percentile on comparable traffic and report the before and after.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Parallelise calls | Cuts serial waits | More load and partial failure handling | The calls depend on each other |
| Stream the response | Better perceived speed | Harder validation before display | The output must be checked first |
| Cache results | Removes repeated work | Staleness and invalidation | Queries are unique |
| Reduce context | Faster prefill | Less evidence for the answer | The task needs the full document |
Good for
Cuts serial waits
Cost
More load and partial failure handling
Fails when
The calls depend on each other
- The calls depend on each other
- The output must be checked first
- Queries are unique
- The task needs the full document
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Parallelise calls is good for Cuts serial waits, costs More load and partial failure handling, and fails when The calls depend on each other. Stream the response is good for Better perceived speed, costs Harder validation before display, and fails when The output must be checked first. Cache results is good for Removes repeated work, costs Staleness and invalidation, and fails when Queries are unique. Reduce context is good for Faster prefill, costs Less evidence for the answer, and fails when The task needs the full document
Failure modes
- Average-chasing. The mean improves while the slow slice worsens. Debug: plot p95 and p99 for the same route before and after the change.
- Retry blindness. A failing dependency is retried and the added time is invisible. Debug: count spans per request and compare with the expected stage count.
- Queue misattribution. Model time looks high because requests wait before prefill. Debug: compare queue wait with prefill time in the same trace.
In production
Keep a per-stage budget that a change must fit, and alert on p95 time to first token and queue depth. Load-test at the concurrency the product actually expects.
Common misconceptions
People think: Caching is the first fix for latency.
Actually: It helps repeated work; if the delay is a serial call or queueing, a cache changes nothing.
People think: A faster model solves a slow app.
Actually: Only when generation dominates the critical path, which a stage breakdown will show.
Interview angle
Strong answers ask for the percentile and the trace, decompose the path, then commit to one measured fix with a cost. Weak answers name a caching layer or a smaller model before measuring.
Sources
- Dean & Barroso, The Tail at Scale — tail latency and why averages mislead.
- Google SRE Book — objectives and latency budgets.
- OpenTelemetry GenAI semantic conventions — stage-level timing attributes.