seniorReportedcurated
When does reranking not help, or actively hurt, a production RAG system?
30-second answer
- If first-stage recall@50 is already near-perfect and the top five is clean, the reranker mostly reorders noise and you pay the latency for nothing.
- If the budget is tight, as in voice, autocomplete or a sub-500 ms p99, the extra model call is the first thing to cut.
- If candidates arrive truncated or stripped of the heading that identifies them, the cross-encoder scores a passage that no longer contains the answer and can demote a correct hit.
- Measure lift as an nDCG@5 delta on your own golden set with retrieval held fixed, so the reranker is the only thing that changed.
- Reranking also cannot fix a missing candidate, so low first-stage recall makes the whole stage an expensive no-op.
Likely follow-ups
Open a question to rehearse the answer.
nDCG@5 improves but answer correctness is flat, what does that tell you about your labels?
- It suggests the judged ranking and answerability are different properties, since topical relevance is not the specific fact the answer needs.
- Check whether the passages that gained position actually contain the claim, and relabel the set with answerability in mind.
- Then check assembly, because correct top-five passages pushed into the middle of a long context change nothing.
How would you route only hard queries to the reranker, and what signal drives the routing?
- Use a signal available before reranking: the score margin between the first and fifth candidates, query length, or a classifier over query type.
- Route when the margin is small, which is exactly where the ordering is unreliable, and skip when one candidate dominates.
- Measure the routed subset's lift separately, or the saving looks free while the hard slice quietly regresses.
What is the smallest rerank depth that keeps the lift, and how do you find it?
- Start from the first-stage recall curve, since the depth where recall flattens is the most the reranker can gain.
- Sweep depth downward and watch nDCG@5, correctness on the labelled set and p95 rerank time, then keep the smallest depth that holds.
- Log the rank at which the winning passage arrived, because a single near-miss at rank nine sets the floor for depth.
Red flags
- assumes always improves
- no isolation
Concepts this tests
Tags
Source: sheldonwangrjt.github.io
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?