Skip to content
AI Engineer Guide.
seniorReportedcurated

When does reranking not help, or actively hurt, a production RAG system?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

nDCG@5 improves but answer correctness is flat, what does that tell you about your labels?
  • It suggests the judged ranking and answerability are different properties, since topical relevance is not the specific fact the answer needs.
  • Check whether the passages that gained position actually contain the claim, and relabel the set with answerability in mind.
  • Then check assembly, because correct top-five passages pushed into the middle of a long context change nothing.
How would you route only hard queries to the reranker, and what signal drives the routing?
  • Use a signal available before reranking: the score margin between the first and fifth candidates, query length, or a classifier over query type.
  • Route when the margin is small, which is exactly where the ordering is unreliable, and skip when one candidate dominates.
  • Measure the routed subset's lift separately, or the saving looks free while the hard slice quietly regresses.
What is the smallest rerank depth that keeps the lift, and how do you find it?
  • Start from the first-stage recall curve, since the depth where recall flattens is the most the reranker can gain.
  • Sweep depth downward and watch nDCG@5, correctness on the labelled set and p95 rerank time, then keep the smallest depth that holds.
  • Log the rank at which the winning passage arrived, because a single near-miss at rank nine sets the floor for depth.

Red flags

  • assumes always improves
  • no isolation

Concepts this tests

Tags

Source: sheldonwangrjt.github.io

More in RAG & retrieval pipelines