midReportedcurated
The vector database already returns similarity scores. Why add a reranker, and how deep do you rerank?
30-second answer
- Vector scores come from a bi-encoder that compressed each document before it ever saw the query, so the vector encodes topicality rather than whether this passage answers this question.
- A cross-encoder reads query and passage together with full attention, which is how it separates the A2 plan from the A2 Lite plan.
- It costs one forward pass per candidate and cannot touch the corpus, so the pipeline is a cascade from ANN over millions down to 5-20 passages.
- Rerank 50-150 candidates down to 5-20, which interactive search can afford at roughly 50-150 ms for the stage, and beyond about 150 candidates latency grows faster than quality.
- Reranking cannot recover evidence the first stage never returned, so fix recall before adding a second model.
Likely follow-ups
Open a question to rehearse the answer.
How do you measure the reranker's lift without confounding it with retrieval changes?
- Hold retrieval fixed and score the same candidate lists with and without the reranker, comparing nDCG@5 and answer correctness.
- Record candidate IDs and rerank scores per request, so a regression can be replayed instead of guessed.
- If ranking improves and answers do not, the bottleneck is assembly or generation rather than the reranker.
How does a late-interaction model like ColBERT compare on storage and latency, and when would you pick it?
- Late interaction keeps a vector per token, so the index is far larger than a single-vector index and memory becomes the constraint.
- It scores with cheap per-token interactions at query time, landing between bi-encoder speed and cross-encoder accuracy.
- Pick it when you need finer matching than a pooled vector but cannot afford a forward pass per candidate.
Your reranker improves nDCG but end-to-end correctness is flat, what do you conclude?
- The ranking metric is not the binding constraint: the evidence was already in the prompt and the generator is losing it.
- Check assembly next, since a correct passage buried in the middle of a long context gets ignored even when ranked first.
- Then check the labels, because a judged ranking that does not reflect answerability can improve while answers do not.
Red flags
- vague "understands context"
- reranks thousands
Concepts this tests
Tags
Source: ishwarj.com
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?