Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 15 of 33

Debug: latency

0 of 33 complete

Why it matters

A slow AI request is a chain of stages: retrieval, reranking, prompt assembly, queueing, prefill, generation and post-processing. Improving the model's tokens per second helps only if generation is the bottleneck. Teams often optimise the stage they understand best and see no change, because the delay sits in a serial call or in queueing behind a concurrency limit. Stage-level timing is what turns a vague complaint into a fix.

Mental model

It is a checkout queue: the total wait is the sum of the queue, the scan and the payment, and speeding up the scan does nothing if the queue is the problem. The metaphor breaks because some stages can run in parallel, so the chain is not always a simple sum.

How it works

Queue wait moves the tail firstInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Modelled latency densities at three concurrency levels, drawn per 500 ms bucket. The service time is fixed at a 700 ms median and the spread comes from queue wait, which rises with in-flight requests. A log-normal shape is assumed; the divergence between the median and the tail is the claim.

What to notice. At thirty-two requests in flight the median is 1,080 ms — 44% above the quiet baseline — but p95 is 4,040 ms against 1,570 ms, and 22.2% of requests pass two seconds against 1.4%. The model's prefill and decode time never changed in this model: queue wait is what grew, and it grows in the tail first. A dashboard showing the median reports a 44% regression where p95 is 157%.

Text version

Latency densities from 0 to 6,000 ms per 500 ms bucket at three concurrency levels. At four requests in flight the curve peaks at 59% per bucket around 750 ms, with a p95 of 1,570 ms and 1.4% of requests over two seconds. At sixteen the peak falls to 37% and moves to 890 ms, p95 is 2,390 ms and 8.9% pass two seconds. At thirty-two the peak is 23% near 1,080 ms, p95 is 4,040 ms and 22.2% pass two seconds. A three-button control selects 4, 16 or 32 in flight; at 16 and 32 the four-in-flight curve stays as a dashed baseline.

  1. Capture a trace of one slow request with a span per stage, including queue and network time.
  2. Compute the latency budget per stage at p50 and p95, not just the total.
  3. Identify the critical path and check whether any stages could run concurrently.
  4. Separate time to first token from inter-token latency, since streaming changes what the user perceives.
  5. Check for retries and fallbacks in the trace, which add a full stage invisibly.
  6. Fix one stage, then re-measure the same percentile on the same traffic mix.

Trade-offs

ChoiceGood forCostFails when
Parallel retrieval callsCuts serial round tripsMore load and harder error handlingThe calls depend on each other's output
Streaming responseBetter perceived latencyHarder to post-process or validateThe output must be checked before display
Smaller modelLower prefill and decode timeQuality loss on hard requestsThe task needs reasoning depth
Aggressive cachingRemoves repeated workStaleness and invalidationQueries are unique
Trade-off mapInteractive

Good for

Cuts serial round trips

Cost

More load and harder error handling

Fails when

The calls depend on each other's output

  1. The calls depend on each other's output
  2. The output must be checked before display
  3. The task needs reasoning depth
  4. Queries are unique

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Parallel retrieval calls is good for Cuts serial round trips, costs More load and harder error handling, and fails when The calls depend on each other's output. Streaming response is good for Better perceived latency, costs Harder to post-process or validate, and fails when The output must be checked before display. Smaller model is good for Lower prefill and decode time, costs Quality loss on hard requests, and fails when The task needs reasoning depth. Aggressive caching is good for Removes repeated work, costs Staleness and invalidation, and fails when Queries are unique

Failure modes

  • Mean-only monitoring. Dashboards look healthy while a subset of users waits seconds. Debug: plot p95 and p99 for the same route and compare with the mean.
  • Hidden retry. A failed call is retried and the trace shows one slow span instead of two. Debug: count model spans per request and compare with the expected one.
  • Queueing misread as model latency. Time to first token grows with concurrency although the model is unchanged. Debug: correlate latency with in-flight request count at the same timestamp.

In production

Alert on p95 time to first token and on queue depth, and keep a per-stage latency budget that a change must fit. Load-test at the concurrency the product actually expects.

Common misconceptions

People think: A faster model fixes a slow application.

Actually: Only if generation is the bottleneck; retrieval, reranking and queueing are frequently larger.

People think: Average latency is a sufficient target.

Actually: Users experience the tail, and a good mean can coexist with a p95 that makes the product unusable.

Interview angle

Decompose the request into stages, name the critical path and the metric you would watch, then choose a fix with its quality cost. Weak answers name a caching layer before measuring where the time goes.

Sources