Why it matters
A bi-encoder compresses each side separately, so any interaction between query and document is lost before comparison. A cross-encoder removes the compression: both texts enter one attention stack and the representation of the query can depend on the passage. That is why it resolves fine distinctions, and why it cannot be indexed — the score exists only for the pair you just ran.
Mental model
A cross-encoder is an interviewer reading the candidate's CV against the job description line by line. A bi-encoder is a recruiter matching two summaries written separately. The interviewer is more accurate and can only see a few candidates a day.
How it works
Synthetic rerank record · query q-8f21
Five candidates · 512-token window · one pass per pair
bounding boxes
Step 1Nothing is compressed
A bi-encoder turns the query and each passage into separate vectors, so the query never sees the passage. The shortlist is what carries that loss forward.
RERANK · QUERY q-8f21 · 5 CANDIDATES SCORED
One forward pass per pair
| Stage 1 | Passage | Tokens |
|---|---|---|
| 1 | p-118 | 412 |
| 2 | p-204 | 388 |
| 3 | p-318 | 461 |
| 4 | p-402 | 296 |
| 5 | p-517 | 540 |
| Passage | Score | Rank | Was |
|---|---|---|---|
| p-318 | 0.88 | 1 | 3 |
| p-517 | 0.79 | 2 | 5 |
| p-204 | 0.44 | 3 | 2 |
| p-118 | 0.31 | 4 | 1 |
| p-402 | 0.12 | 5 | 4 |
| Score | Reading | Pairs |
|---|---|---|
| 0.79 – 0.88 | answers the question | 2 |
| 0.31 – 0.44 | related, not an answer | 2 |
| 0.12 | off-topic | 1 |
5 pairs · 1 over window · 2 below 0.50 · 1 pass per pair · 0 index built
Step record
- Retrieved
- 50
- Scored here
- 5
- Indexed
- bi-encoder only
The text version summarises this figure; select a region for its full text.
Figure. One query and the five candidates a bi-encoder passed on. The cross-encoder scores each pair jointly and reorders the list: p-318 moves from third to first, p-517 from fifth to second, and the weakest candidate falls. p-517 is also 540 tokens against a 512 window, so its last 28 tokens never reached the score.
What to notice. Synthetic rerank record. Scores are authored to show the reordering and are not probabilities; the floor and the bands are labelled, not measured. The figure shows why cost scales with candidates and what truncation removes, not a model's accuracy.
Text version
A rerank record for the query “Can I cancel an annual plan after it renews and still get a refund?” A stage-one table lists five candidates with token counts: p-118 at 412, p-204 at 388, p-318 at 461, p-402 at 296 and p-517 at 540. A cross-encoder table gives p-318 0.88 and rank 1 from rank 3, p-517 0.79 and rank 2 from rank 5, p-204 0.44 rank 3 from 2, p-118 0.31 rank 4 from 1 and p-402 0.12 rank 5 from 4. Fields give a 512-token window, a 0.50 floor and one pass per pair. A note records that p-517 loses its last 28 tokens. Score bands group two high, two middle and one low score. Steps mark the shortlist, the joint scoring, the reorder, truncation, the floor and the cost.
- Collect training pairs where the label reflects answerability, not just topical overlap, and include hard negatives from a first-stage retriever.
- Fine-tune a small pretrained encoder with a relevance head, using a binary or pairwise loss over query-passage pairs.
- Truncate both sides deliberately: the passage window is the model's real input, so log any candidate that exceeds it.
- Score candidates in batches sized to the latency budget, and sort by score rather than by the first-stage order.
- Calibrate a score floor on labelled data so the model can trigger abstention rather than always returning a top hit.
- Distil the cross-encoder into a smaller model or a bi-encoder if per-query cost becomes the binding constraint.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Cross-encoder reranking | Precise ordering of a shortlist | One forward pass per candidate | The candidate list is thousands long |
| Bi-encoder only | Indexable and cheap at query time | Loses query-document interaction | Queries hinge on a qualifier or negation |
| Late interaction | Cheap-ish multi-vector matching | Larger index and more storage | Memory is tightly constrained |
| Distilled cross-encoder | Lower latency per candidate | Some accuracy loss on hard pairs | The task needs maximum precision |
Good for
Precise ordering of a shortlist
Cost
One forward pass per candidate
Fails when
The candidate list is thousands long
- The candidate list is thousands long
- Queries hinge on a qualifier or negation
- Memory is tightly constrained
- The task needs maximum precision
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Cross-encoder reranking is good for Precise ordering of a shortlist, costs One forward pass per candidate, and fails when The candidate list is thousands long. Bi-encoder only is good for Indexable and cheap at query time, costs Loses query-document interaction, and fails when Queries hinge on a qualifier or negation. Late interaction is good for Cheap-ish multi-vector matching, costs Larger index and more storage, and fails when Memory is tightly constrained. Distilled cross-encoder is good for Lower latency per candidate, costs Some accuracy loss on hard pairs, and fails when The task needs maximum precision
Failure modes
- Candidate truncation. The passage is cut before the decisive sentence and the score reflects the wrong text. Debug: log token counts of scored pairs and inspect the ones at the limit.
- Training distribution mismatch. The model was trained on web question-answer pairs and scores your technical passages poorly. Debug: compare its ordering with human judgements on a sample of your own queries.
- Score used as a probability. A raw logit is treated as confidence and thresholds are set on it. Debug: plot score distributions for relevant and irrelevant pairs before choosing a cut-off.
In production
Track rerank latency per candidate and the score distribution over time. Pin the model version, since a new checkpoint silently changes the ordering of every query.
Common misconceptions
People think: A cross-encoder can replace the vector index.
Actually: It scores pairs, so using it over a corpus is a linear scan with a transformer per document.
People think: Cross-encoder scores are comparable across models.
Actually: Each checkpoint has its own scale, so thresholds must be recalibrated when the model changes.
Interview angle
Explain joint versus separate encoding as information loss, then derive the cost model: one pass per pair, hence a shortlist. Mention truncation and score calibration. Calling it a better embedding model, and failing to explain why it cannot be indexed, is the standard error.
Sources
- Nogueira & Cho, Passage Re-ranking with BERT — the canonical cross-encoder formulation for passage ranking.
- Sentence Transformers, Retrieve and re-rank — a two-stage retrieve-then-cross-encode pipeline in practice.
- Khattab & Zaharia, ColBERT — late interaction as the cost compromise between the two extremes.