Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 35 of 57

Cross-encoders

0 of 57 complete

Why it matters

A bi-encoder compresses each side separately, so any interaction between query and document is lost before comparison. A cross-encoder removes the compression: both texts enter one attention stack and the representation of the query can depend on the passage. That is why it resolves fine distinctions, and why it cannot be indexed — the score exists only for the pair you just ran.

Mental model

A cross-encoder is an interviewer reading the candidate's CV against the job description line by line. A bi-encoder is a recruiter matching two summaries written separately. The interviewer is more accurate and can only see a few candidates a day.

How it works

Five pairs, five score recordsInteractive

Synthetic rerank record · query q-8f21

Five candidates · 512-token window · one pass per pair

bounding boxes

Step 1/6 · Nothing is compressed

Step 1Nothing is compressed

A bi-encoder turns the query and each passage into separate vectors, so the query never sees the passage. The shortlist is what carries that loss forward.

RERANK · QUERY q-8f21 · 5 CANDIDATES SCORED

, running header, region 1. Select to inspect.

One forward pass per pair

, title, region 2. Select to inspect.
Query

Can I cancel an annual plan after it renews and still get a refund?

, paragraph Query, region 3. Select to inspect.
SHORTLIST
Stage 1PassageTokens
1p-118412
2p-204388
3p-318461
4p-402296
5p-517540
, table SHORTLIST, region 4. Select to inspect.
CROSS SCORES
PassageScoreRankWas
p-3180.8813
p-5170.7925
p-2040.4432
p-1180.3141
p-4020.1254
, table CROSS SCORES, region 5. Select to inspect.
Window512 tokens
, form field Window, region 6. Select to inspect.
Floor0.50
, form field Floor, region 7. Select to inspect.
Cost1 pass per pair
, form field Cost, region 8. Select to inspect.
Truncation

p-517 is 540 tokens: the last 28 are cut before scoring, so a sentence past the window cannot change its rank.

, paragraph Truncation, region 9. Select to inspect.
BANDS
ScoreReadingPairs
0.79 – 0.88answers the question2
0.31 – 0.44related, not an answer2
0.12off-topic1
, table BANDS, region 10. Select to inspect.

5 pairs · 1 over window · 2 below 0.50 · 1 pass per pair · 0 index built

, running footer, region 11. Select to inspect.
11 regions

Step record

Retrieved
50
Scored here
5
Indexed
bi-encoder only

The text version summarises this figure; select a region for its full text.

Figure. One query and the five candidates a bi-encoder passed on. The cross-encoder scores each pair jointly and reorders the list: p-318 moves from third to first, p-517 from fifth to second, and the weakest candidate falls. p-517 is also 540 tokens against a 512 window, so its last 28 tokens never reached the score.

What to notice. Synthetic rerank record. Scores are authored to show the reordering and are not probabilities; the floor and the bands are labelled, not measured. The figure shows why cost scales with candidates and what truncation removes, not a model's accuracy.

Text version

A rerank record for the query “Can I cancel an annual plan after it renews and still get a refund?” A stage-one table lists five candidates with token counts: p-118 at 412, p-204 at 388, p-318 at 461, p-402 at 296 and p-517 at 540. A cross-encoder table gives p-318 0.88 and rank 1 from rank 3, p-517 0.79 and rank 2 from rank 5, p-204 0.44 rank 3 from 2, p-118 0.31 rank 4 from 1 and p-402 0.12 rank 5 from 4. Fields give a 512-token window, a 0.50 floor and one pass per pair. A note records that p-517 loses its last 28 tokens. Score bands group two high, two middle and one low score. Steps mark the shortlist, the joint scoring, the reorder, truncation, the floor and the cost.

  1. Collect training pairs where the label reflects answerability, not just topical overlap, and include hard negatives from a first-stage retriever.
  2. Fine-tune a small pretrained encoder with a relevance head, using a binary or pairwise loss over query-passage pairs.
  3. Truncate both sides deliberately: the passage window is the model's real input, so log any candidate that exceeds it.
  4. Score candidates in batches sized to the latency budget, and sort by score rather than by the first-stage order.
  5. Calibrate a score floor on labelled data so the model can trigger abstention rather than always returning a top hit.
  6. Distil the cross-encoder into a smaller model or a bi-encoder if per-query cost becomes the binding constraint.

Trade-offs

ChoiceGood forCostFails when
Cross-encoder rerankingPrecise ordering of a shortlistOne forward pass per candidateThe candidate list is thousands long
Bi-encoder onlyIndexable and cheap at query timeLoses query-document interactionQueries hinge on a qualifier or negation
Late interactionCheap-ish multi-vector matchingLarger index and more storageMemory is tightly constrained
Distilled cross-encoderLower latency per candidateSome accuracy loss on hard pairsThe task needs maximum precision
Trade-off mapInteractive

Good for

Precise ordering of a shortlist

Cost

One forward pass per candidate

Fails when

The candidate list is thousands long

  1. The candidate list is thousands long
  2. Queries hinge on a qualifier or negation
  3. Memory is tightly constrained
  4. The task needs maximum precision

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Cross-encoder reranking is good for Precise ordering of a shortlist, costs One forward pass per candidate, and fails when The candidate list is thousands long. Bi-encoder only is good for Indexable and cheap at query time, costs Loses query-document interaction, and fails when Queries hinge on a qualifier or negation. Late interaction is good for Cheap-ish multi-vector matching, costs Larger index and more storage, and fails when Memory is tightly constrained. Distilled cross-encoder is good for Lower latency per candidate, costs Some accuracy loss on hard pairs, and fails when The task needs maximum precision

Failure modes

  • Candidate truncation. The passage is cut before the decisive sentence and the score reflects the wrong text. Debug: log token counts of scored pairs and inspect the ones at the limit.
  • Training distribution mismatch. The model was trained on web question-answer pairs and scores your technical passages poorly. Debug: compare its ordering with human judgements on a sample of your own queries.
  • Score used as a probability. A raw logit is treated as confidence and thresholds are set on it. Debug: plot score distributions for relevant and irrelevant pairs before choosing a cut-off.

In production

Track rerank latency per candidate and the score distribution over time. Pin the model version, since a new checkpoint silently changes the ordering of every query.

Common misconceptions

People think: A cross-encoder can replace the vector index.

Actually: It scores pairs, so using it over a corpus is a linear scan with a transformer per document.

People think: Cross-encoder scores are comparable across models.

Actually: Each checkpoint has its own scale, so thresholds must be recalibrated when the model changes.

Interview angle

Explain joint versus separate encoding as information loss, then derive the cost model: one pass per pair, hence a shortlist. Mention truncation and score calibration. Calling it a better embedding model, and failing to explain why it cannot be indexed, is the standard error.

Sources