Why it matters
A slow AI request is a chain of stages: retrieval, reranking, prompt assembly, queueing, prefill, generation and post-processing. Improving the model's tokens per second helps only if generation is the bottleneck. Teams often optimise the stage they understand best and see no change, because the delay sits in a serial call or in queueing behind a concurrency limit. Stage-level timing is what turns a vague complaint into a fix.
Mental model
It is a checkout queue: the total wait is the sum of the queue, the scan and the payment, and speeding up the scan does nothing if the queue is the problem. The metaphor breaks because some stages can run in parallel, so the chain is not always a simple sum.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Modelled latency densities at three concurrency levels, drawn per 500 ms bucket. The service time is fixed at a 700 ms median and the spread comes from queue wait, which rises with in-flight requests. A log-normal shape is assumed; the divergence between the median and the tail is the claim.
What to notice. At thirty-two requests in flight the median is 1,080 ms — 44% above the quiet baseline — but p95 is 4,040 ms against 1,570 ms, and 22.2% of requests pass two seconds against 1.4%. The model's prefill and decode time never changed in this model: queue wait is what grew, and it grows in the tail first. A dashboard showing the median reports a 44% regression where p95 is 157%.
Text version
Latency densities from 0 to 6,000 ms per 500 ms bucket at three concurrency levels. At four requests in flight the curve peaks at 59% per bucket around 750 ms, with a p95 of 1,570 ms and 1.4% of requests over two seconds. At sixteen the peak falls to 37% and moves to 890 ms, p95 is 2,390 ms and 8.9% pass two seconds. At thirty-two the peak is 23% near 1,080 ms, p95 is 4,040 ms and 22.2% pass two seconds. A three-button control selects 4, 16 or 32 in flight; at 16 and 32 the four-in-flight curve stays as a dashed baseline.
- Capture a trace of one slow request with a span per stage, including queue and network time.
- Compute the latency budget per stage at p50 and p95, not just the total.
- Identify the critical path and check whether any stages could run concurrently.
- Separate time to first token from inter-token latency, since streaming changes what the user perceives.
- Check for retries and fallbacks in the trace, which add a full stage invisibly.
- Fix one stage, then re-measure the same percentile on the same traffic mix.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Parallel retrieval calls | Cuts serial round trips | More load and harder error handling | The calls depend on each other's output |
| Streaming response | Better perceived latency | Harder to post-process or validate | The output must be checked before display |
| Smaller model | Lower prefill and decode time | Quality loss on hard requests | The task needs reasoning depth |
| Aggressive caching | Removes repeated work | Staleness and invalidation | Queries are unique |
Good for
Cuts serial round trips
Cost
More load and harder error handling
Fails when
The calls depend on each other's output
- The calls depend on each other's output
- The output must be checked before display
- The task needs reasoning depth
- Queries are unique
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Parallel retrieval calls is good for Cuts serial round trips, costs More load and harder error handling, and fails when The calls depend on each other's output. Streaming response is good for Better perceived latency, costs Harder to post-process or validate, and fails when The output must be checked before display. Smaller model is good for Lower prefill and decode time, costs Quality loss on hard requests, and fails when The task needs reasoning depth. Aggressive caching is good for Removes repeated work, costs Staleness and invalidation, and fails when Queries are unique
Failure modes
- Mean-only monitoring. Dashboards look healthy while a subset of users waits seconds. Debug: plot p95 and p99 for the same route and compare with the mean.
- Hidden retry. A failed call is retried and the trace shows one slow span instead of two. Debug: count model spans per request and compare with the expected one.
- Queueing misread as model latency. Time to first token grows with concurrency although the model is unchanged. Debug: correlate latency with in-flight request count at the same timestamp.
In production
Alert on p95 time to first token and on queue depth, and keep a per-stage latency budget that a change must fit. Load-test at the concurrency the product actually expects.
Common misconceptions
People think: A faster model fixes a slow application.
Actually: Only if generation is the bottleneck; retrieval, reranking and queueing are frequently larger.
People think: Average latency is a sufficient target.
Actually: Users experience the tail, and a good mean can coexist with a p95 that makes the product unusable.
Interview angle
Decompose the request into stages, name the critical path and the metric you would watch, then choose a fix with its quality cost. Weak answers name a caching layer before measuring where the time goes.
Sources
- Dean & Barroso, The Tail at Scale — why tail latency, not the mean, defines user experience.
- Google SRE Book — objectives, budgets and latency as a service property.
- OpenTelemetry GenAI semantic conventions — timing attributes for model and tool spans.