seniorReportedcurated
After adding query rewriting and a reranker, p95 went from 900 ms to 4 s. Where do you look and what do you cut?
30-second answer
- Break p95 into per-stage spans before changing anything: rewrite 100-300 ms, hybrid retrieval 50-100 ms, rerank 50-150 ms, assembly 20 ms, so 220-570 ms sits before generation and cannot explain four seconds.
- When the traced spans sum well under a second, the missing time is unattributed: extra model calls the trace does not span, queueing on a shared inference endpoint, or retries after timeouts and rate limits.
- Measure that gap per request, carrying queue wait, retry count and output tokens, because summed stage spans and end-to-end p95 are answering different questions.
- Cheapest wins are skipping the rewrite when there is no conversation history, running it in parallel with retrieving the raw query, caching query embeddings, cutting rerank depth from 150 to 50, and streaming.
- Running the rewrite in parallel and caching the query embedding remove duplicated work at no quality cost, while skipping a rewrite that had history and halving rerank depth trade quality, so those need the guardrail.
- None of them touches queue wait or the decode loop on a shared endpoint, so a stage budget cannot fix a tail that lives outside the stages you own.
Likely follow-ups
Open a question to rehearse the answer.
Which of those cuts would you A/B first, and what guardrail metric tells you it hurt quality?
- Start with the cut that removes work rather than quality: run the rewrite concurrently with the raw query, and cache query embeddings so repeats do not pay twice.
- Guard on final-context recall@k and answer correctness on the labelled set, with abstain rate as the early signal, since it moves before quality does.
- Cut rerank depth last, because depth is what recovers the near-miss passage the first stage ranked ninth.
Streaming hides latency from the user but not from the queue, how do you report perceived versus total p95?
- Report time to first token separately from total duration, because streaming changes when output appears and not how long generation takes.
- Keep a server-side total p95 for capacity and cost, and a client-side first-token p95 for the experience, measured at both boundaries so proxy buffering shows up.
- A response that shows a token in 400 ms but runs eight seconds still holds a queue slot, so the total is what sizes replicas.
The per-stage traces sum to 1.2 s but p95 is 4 s, what do you measure next?
- Measure queue wait and admission delay, which sit before the first span and are the component that collapses under load.
- Count spans per request against the expected stage count, because a retry or a second rewrite pass adds time no stage budget sees.
- Compare the client first-byte timestamp with the origin's, since a buffering proxy keeps server-side timings healthy while users wait.
Red flags
- smaller model first
- no instrumentation
Concepts this tests
Tags
Source: www.mockingly.ai
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?