Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 20 of 22

Cross-modal retrieval

0 of 22 complete

Why it matters

A user who wants a photo of a specific thing will not have the photo to search with. Cross-modal retrieval makes query and corpus comparable by projecting both into one space. The difficulty is that the space is learned from paired data, so it reflects the captions in that data: rare concepts, spatial relations and negation are represented poorly, and no index tuning fixes a missing direction.

Mental model

Cross-modal retrieval is a phrasebook shared by two languages: both sides are translated into it, and meaning survives only where the book has entries. The metaphor breaks because the space is continuous, so a concept the book lacks can still land near something plausible and wrong.

How it works

Mechanism, step by stepInteractive
  1. 1Pick an embedding model trained contrastively on the modality pair you need, and check the languages and domains of its training data.
  2. 2Normalise embeddings and fix one similarity function across indexing and querying.
  3. 3Index at the granularity you will retrieve, such as whole image, region or clip, since one vector cannot represent all three.
  4. 4Evaluate with text queries written by users, not with captions reused from the training distribution.
  5. 5Measure recall at k on hard negatives: same object with a different attribute, and same scene with a different relation.
  6. 6Add a reranker in the target modality when precision matters more than latency.

Step 1 of 6

Pick an embedding model trained contrastively on the modality pair you need, and check the languages and domains of its training data.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Pick an embedding model trained contrastively on the modality pair you need, and check th… 2. Normalise embeddings and fix one similarity function across indexing and querying. 3. Index at the granularity you will retrieve, such as whole image, region or clip, since on… 4. Evaluate with text queries written by users, not with captions reused from the training d… 5. Measure recall at k on hard negatives: same object with a different attribute, and same s… 6. Add a reranker in the target modality when precision matters more than latency.

  1. Pick an embedding model trained contrastively on the modality pair you need, and check the languages and domains of its training data.
  2. Normalise embeddings and fix one similarity function across indexing and querying.
  3. Index at the granularity you will retrieve, such as whole image, region or clip, since one vector cannot represent all three.
  4. Evaluate with text queries written by users, not with captions reused from the training distribution.
  5. Measure recall at k on hard negatives: same object with a different attribute, and same scene with a different relation.
  6. Add a reranker in the target modality when precision matters more than latency.

Trade-offs

ChoiceGood forCostFails when
Single joint spaceOne index, low latencyCompositional queries failQueries combine attributes and relations
Per-modality encoders plus translationReuses strong unimodal modelsTranslation loss and extra latencyQueries are abstract
Region or clip embeddingsFiner retrieval granularityMore vectors and storageThe query is about the whole item
Rerank with a cross-modal modelBetter precisionLatency per candidateYou need many results per query
Trade-off mapInteractive

Good for

One index, low latency

Cost

Compositional queries fail

Fails when

Queries combine attributes and relations

  1. Queries combine attributes and relations
  2. Queries are abstract
  3. The query is about the whole item
  4. You need many results per query

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Single joint space is good for One index, low latency, costs Compositional queries fail, and fails when Queries combine attributes and relations. Per-modality encoders plus translation is good for Reuses strong unimodal models, costs Translation loss and extra latency, and fails when Queries are abstract. Region or clip embeddings is good for Finer retrieval granularity, costs More vectors and storage, and fails when The query is about the whole item. Rerank with a cross-modal model is good for Better precision, costs Latency per candidate, and fails when You need many results per query

Failure modes

  • Attribute binding failure. A query for a red cube returns blue cubes because colour and shape are not bound together. Debug: build a probe set that varies one attribute at a time and measure accuracy per combination.
  • Caption distribution bias. Queries phrased unlike the training captions retrieve poorly. Debug: compare recall for user-written queries against captions from the same corpus.
  • Normalisation mismatch. Index and query vectors are normalised differently and rankings invert. Debug: assert unit norm on both paths and compare top results for a fixed query.

In production

Track zero-result and low-score query rates as a coverage signal, and sample them weekly. Version the embedding model with the index, because a swap invalidates every stored vector.

Common misconceptions

People think: Cross-modal retrieval understands images.

Actually: It matches distributions of paired text and pixels; understanding is an inference you make from the matches.

People think: Better indexes fix poor recall.

Actually: Approximate search preserves the ranking it is given; a missing direction in the embedding space stays missing.

Interview angle

The shared space is the whole story: how it is trained and why compositional queries break it. A strong answer proposes a probe set that varies one attribute at a time. Weak answers describe a vector database and never mention that recall is bounded by the paired data the encoder saw.

Sources