Skip to content
AI Engineer Guide.
seniorReportedcurated

After adding query rewriting and a reranker, p95 went from 900 ms to 4 s. Where do you look and what do you cut?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Which of those cuts would you A/B first, and what guardrail metric tells you it hurt quality?
  • Start with the cut that removes work rather than quality: run the rewrite concurrently with the raw query, and cache query embeddings so repeats do not pay twice.
  • Guard on final-context recall@k and answer correctness on the labelled set, with abstain rate as the early signal, since it moves before quality does.
  • Cut rerank depth last, because depth is what recovers the near-miss passage the first stage ranked ninth.
Streaming hides latency from the user but not from the queue, how do you report perceived versus total p95?
  • Report time to first token separately from total duration, because streaming changes when output appears and not how long generation takes.
  • Keep a server-side total p95 for capacity and cost, and a client-side first-token p95 for the experience, measured at both boundaries so proxy buffering shows up.
  • A response that shows a token in 400 ms but runs eight seconds still holds a queue slot, so the total is what sizes replicas.
The per-stage traces sum to 1.2 s but p95 is 4 s, what do you measure next?
  • Measure queue wait and admission delay, which sit before the first span and are the component that collapses under load.
  • Count spans per request against the expected stage count, because a retry or a second rewrite pass adds time no stage budget sees.
  • Compare the client first-byte timestamp with the origin's, since a buffering proxy keeps server-side timings healthy while users wait.

Red flags

  • smaller model first
  • no instrumentation

Concepts this tests

Tags

Source: www.mockingly.ai

More in RAG & retrieval pipelines