Why retrieval alone is not enough
A vector database returns neighbours by similarity, and similarity is not the same as answering the question. The first-stage retriever is usually a bi-encoder: it turns each passage into an embedding before it ever sees the query. That design is what makes search over millions of passages feasible, and it is also the bottleneck. Two passages can be equally close to the query while only one contains the fact that resolves it.
The classic failure is topical. Ask "does the A2 plan include priority support" and a chunk describing the A2 Lite plan can rank first, because both chunks are about the same product family. The embedding captures the topic; it does not capture the distinction. Reranking exists to fix that gap before the context window is filled with almost-right evidence.
Mental model: a second interview
The first stage screens thousands of CVs on keyword overlap; the reranker interviews the shortlist and reads each candidate against the actual role. Screening is cheap and broad, interviewing is expensive and precise.
The analogy breaks in two places. A reranker scores each candidate independently and never compares candidates with each other, so it cannot enforce diversity; and it has no memory of the query beyond the text you give it. If the shortlist is wrong, the interview cannot fix it.
How a reranker works
A cross-encoder takes the query and one candidate passage together, runs them through a transformer, and outputs a relevance score computed independently for each candidate. Because full attention runs across both texts, the model can weigh the exact phrase, the negation, and the qualifier that a pooled vector lost.
That accuracy has a price: one forward pass per candidate, and no reusable index. So the pipeline is a cascade:
- Retrieve 50–150 candidates with hybrid search.
- Score each candidate with the cross-encoder.
- Keep the top 5–20 by reranked score.
- Assemble those passages into the prompt in a deliberate order.
The scoring step is usually a small model, so the cost is tens of milliseconds per batch rather than per document. If a candidate scores poorly, that is useful signal: it can trigger an abstain path instead of an answer.
Choosing the depth
Rerank depth is a budget decision. Start from the first-stage recall curve: if recall@50 is 0.97 and recall@150 is 0.98, the extra hundred candidates buy almost nothing and cost linearly. If recall@50 is 0.82, deeper retrieval is the cheaper fix, even before touching the reranker.
The second input is the latency budget. Interactive search can afford 50–150 ms. Voice and autocomplete usually cannot, so they either rerank fewer candidates or skip the stage entirely. A useful compromise is conditional reranking: run it only when the retrieval margin is small or the query is long.
When reranking hurts
Three cases are worth naming:
- The top of the list is already clean. If first-stage precision is high and the downstream answer is correct, the reranker is reordering noise and adding latency.
- Candidates arrive truncated. A reranker can only score what it sees. If chunk assembly cut the heading that identifies the section, the cross-encoder may demote the correct hit.
- The metric is wrong. A reranker can improve nDCG without improving answer correctness. Measure the end-to-end outcome, not just the ranking metric.
A minimal implementation
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query: str, candidates: list[dict], keep: int = 8) -> list[dict]:
pairs = [(query, c["text"]) for c in candidates]
scores = reranker.predict(pairs, batch_size=32)
for candidate, score in zip(candidates, scores):
candidate["rerank_score"] = float(score)
return sorted(candidates, key=lambda c: c["rerank_score"], reverse=True)[:keep]The model name matters less than the evaluation. Pin the version, record the candidates and scores per query, and compare against a baseline that skips the stage.
Text version
A query retrieves a wide top-k, a cross-encoder scores each query-document pair jointly, the scores are sorted, and only the top-n enter the prompt.
Query
does the A2 plan include priority support?
- Candidates scored
- 0 of 12
- Forward passes
- 0
First-stage candidates
ordered by bi-encoder similarity
- 1A2 Lite plan overviewtopicalb 0.91
- 2All plans comparedb 0.89
- 3A2 plan inclusionsb 0.87
- 4Priority support SLAb 0.84
- 5Support channelsb 0.82
- 6Upgrade A1 → A2b 0.80
- 7A2 Lite FAQb 0.78
- 8Billing and invoicesb 0.76
- 9Priority support from A2 upnear-missb 0.74
- 10Enterprise support add-onb 0.71
- 11Status page incidentsb 0.69
- 12Contact salesb 0.66
cross-encoder (x)bi-encoder (b)not scored at this depth
Prompt window
Nothing selected. Rescore the candidates, then keep the top four.
The first stage returns everything about the plan family, including the A2 Lite page. Similarity ranks topic, not answerability.
Illustrative scores, not a benchmark. Latency is modelled at 8 ms per candidate on top of 62 ms of retrieval; real cross-encoders batch, so cost per candidate falls with batch size.
Figure. A first-stage retriever returns twelve passages ordered by embedding similarity; a cross-encoder rescores the top d of them and the best four enter the prompt. Four slots are used here for legibility; the page recommends 5–20. Depth is a budget decision: every extra candidate is another forward pass, and only candidates inside the depth can be rescued. Scores and the cost model are illustrative, and real cross-encoders batch.
What to notice. The passage that answers the question arrives at first-stage rank 9. At depth 8 it is never scored and cannot reach the prompt; at depth 9 it moves to rank 2, while the topical A2 Lite page falls from rank 1 to rank 7, and to rank 8 at full depth. Reranking reorders what was retrieved — it cannot recover a passage the first stage never returned.
Text version
The query is 'does the A2 plan include priority support'. Twelve candidates arrive with bi-encoder scores from 0.91 down to 0.66, starting with the A2 Lite plan overview, all plans compared, A2 plan inclusions, priority support SLA, support channels, upgrade from A1 to A2, A2 Lite FAQ, billing, priority support from A2 up, enterprise support add-on, status page and contact sales. A depth slider is set to 12 of 12; after rescoring, the readout shows twelve of twelve candidates scored. After rescoring, the order is: A2 plan inclusions 0.95, priority support from A2 up 0.93 moving up seven places from rank 9 and labelled as the near miss, priority support SLA 0.71, enterprise support add-on 0.68, upgrade A1 to A2 0.52, all plans compared 0.44, support channels 0.35, A2 Lite plan overview 0.31 falling seven places from rank 1 and labelled topical, A2 Lite FAQ 0.29, billing 0.21, contact sales 0.18, status page 0.12. Each row carries a bar proportional to its score. The prompt window then lists the top four: A2 plan inclusions, priority support from A2 up, priority support SLA and enterprise support add-on. Lowering the depth to 8 leaves the near miss unscored at rank 9, outside the four selected passages; the readout drops to eight of twelve candidates scored.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| No reranker | Low latency, clean top-k | Misses near-miss topical matches | Queries need fine distinctions |
| Cross-encoder, depth 50 | Most interactive products | 50–100 ms, one model | First-stage recall is already the problem |
| Cross-encoder, depth 150+ | Hard, long-tail queries | Latency and GPU cost | Budget is tight or candidates are truncated |
| LLM-based listwise rerank | Complex multi-constraint queries | Large token cost, unstable ordering | You need predictable p99 |
What to measure
Report recall before and after the stage, nDCG@5 on a labelled set, and answer correctness with retrieval held fixed. Then watch p95 latency and cost per query. If the reranker improves ranking but not answers, the bottleneck is elsewhere — often context assembly or the generator.
The honest summary: reranking is a precision tool with a real latency bill. Add it when the failure mode is "the right passage was there but ranked sixth", not when it was never retrieved.
In production
- Budget the stage separately. Track rerank latency apart from retrieval and generation; a p95 that doubles is usually depth creep, not model drift.
- Batch and cache. Score candidates in one batch and cache scores keyed by
(query_hash, chunk_id, model_version); invalidate on model change. - Record scores and versions. Store candidate IDs,
rerank_scoreand model version per request so a regression can be replayed. - Fail open. If the reranker times out, fall back to the first-stage order and mark the response as degraded rather than failing the request.
Common misconceptions
People think: A reranker makes retrieval more accurate for every query.
Actually: It improves the ordering of what was retrieved; it cannot recover a passage the first stage missed.
People think: Reranking and a better embedding model are interchangeable.
Actually: They fix different failures: embedding quality changes recall; reranking changes precision among candidates.
People think: Deeper reranking is always better.
Actually: Beyond roughly 150 candidates, latency grows faster than quality; useful depth is bounded by first-stage recall and the latency budget.
Failure modes
- Reranking a missing candidate. If the first stage never retrieved the passage, the reranker cannot recover it. Measure recall@50 before changing the model; if it is low, fix retrieval first.
- Truncated candidates. A candidate scored on a fragment that lost its heading can be demoted incorrectly. Log the exact text sent to the reranker for a failing query.
- Latency creep. Depth grows quietly during tuning until p95 doubles. Alert on rerank time separately from total retrieval time.
- Metric myopia. nDCG improves while answer correctness stays flat. Hold retrieval fixed and compare end-to-end answers, not only ranking.
Interview angle
Expect "why not just use a better embedding model?" Answer with the bi-encoder bottleneck: the document vector is computed without the query, so it encodes topic rather than answerability. Then name the cost: one forward pass per candidate, which is why you rerank a shortlist and never the corpus. A strong answer includes the depth trade-off (50–150), the latency bill, and the measurement that proves lift. See RAG evaluation and the RAG interview category.
Related
Sources
- Nogueira & Cho, Passage Re-ranking with BERT — the canonical cross-encoder formulation.
- Sentence Transformers: retrieve and re-rank — practical two-stage pipeline.
- Khattab & Zaharia, ColBERT — late-interaction alternative.