Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 30 of 33

Troubleshoot: slow app

0 of 33 complete

Why it matters

Latency complaints arrive as a single number, usually an average, and the work is to turn that into a stage budget. Under drill conditions the interviewer watches whether you measure before changing and whether you can name the trade-off in your fix. Adding a cache or a smaller model without a stage breakdown is a guess that sometimes helps and cannot be defended. The strong answer is a short diagnostic sequence with one change and a re-measurement.

Mental model

It is a relay race with a stopwatch at each handover: the total is the sum, and the slowest leg sets the floor. The metaphor breaks because legs can sometimes run in parallel, so the critical path is not always the longest single stage.

How it works

A median win can be a tail lossInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Cumulative latency distributions for one route before and after a single change: the share of requests finished at or below each latency. The log-normal models share a family but not a tail, and the cache curve crosses the baseline near 1,500 ms — faster below it, slower above. Constants are illustrative.

What to notice. The cache fix moves the median from 900 to 750 ms — a 17% win on the headline number — while p95 rises from 2,220 to 2,580 ms and p99 from 3,240 to 4,290 ms. The curves cross near 1,500 ms: every request below the crossing is faster and the slow slice is worse. Parallel retrieval moves the whole curve: median 620 ms, p95 1,370 ms.

Text version

Cumulative share of requests served against latency from 0 to 4,000 ms. The baseline runs from a 900 ms median to 2,220 ms at p95 and 3,240 ms at p99. After caching, the curve is above the baseline until about 1,500 ms and below it after: median 750 ms, p95 2,580 ms, p99 4,290 ms. After parallel retrieval the whole curve shifts left: median 620 ms and p95 1,370 ms. A three-button control selects no change, caching or parallel retrieval, a vertical line marks an example 2,000 ms budget, and a panel reports p50, p95 and p99 for the selected frame.

  1. Ask for p50 and p95, the route, and whether the complaint is about first token or full response.
  2. Request a trace with one span per stage, including retrieval, reranking, queue wait and generation.
  3. Compute the share of total time per stage and identify the critical path.
  4. Check for serial calls that could run concurrently and for retries that duplicate a stage.
  5. Propose one change with its expected effect and its quality or cost.
  6. Re-measure the same percentile on comparable traffic and report the before and after.

Trade-offs

ChoiceGood forCostFails when
Parallelise callsCuts serial waitsMore load and partial failure handlingThe calls depend on each other
Stream the responseBetter perceived speedHarder validation before displayThe output must be checked first
Cache resultsRemoves repeated workStaleness and invalidationQueries are unique
Reduce contextFaster prefillLess evidence for the answerThe task needs the full document
Trade-off mapInteractive

Good for

Cuts serial waits

Cost

More load and partial failure handling

Fails when

The calls depend on each other

  1. The calls depend on each other
  2. The output must be checked first
  3. Queries are unique
  4. The task needs the full document

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Parallelise calls is good for Cuts serial waits, costs More load and partial failure handling, and fails when The calls depend on each other. Stream the response is good for Better perceived speed, costs Harder validation before display, and fails when The output must be checked first. Cache results is good for Removes repeated work, costs Staleness and invalidation, and fails when Queries are unique. Reduce context is good for Faster prefill, costs Less evidence for the answer, and fails when The task needs the full document

Failure modes

  • Average-chasing. The mean improves while the slow slice worsens. Debug: plot p95 and p99 for the same route before and after the change.
  • Retry blindness. A failing dependency is retried and the added time is invisible. Debug: count spans per request and compare with the expected stage count.
  • Queue misattribution. Model time looks high because requests wait before prefill. Debug: compare queue wait with prefill time in the same trace.

In production

Keep a per-stage budget that a change must fit, and alert on p95 time to first token and queue depth. Load-test at the concurrency the product actually expects.

Common misconceptions

People think: Caching is the first fix for latency.

Actually: It helps repeated work; if the delay is a serial call or queueing, a cache changes nothing.

People think: A faster model solves a slow app.

Actually: Only when generation dominates the critical path, which a stage breakdown will show.

Interview angle

Strong answers ask for the percentile and the trace, decompose the path, then commit to one measured fix with a cost. Weak answers name a caching layer or a smaller model before measuring.

Sources