Skip to content
AI Engineer Guide.
midReportedcurated

Where do embeddings systematically fail as a retrieval mechanism, and which failures can you fix cheaply?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How would you build a golden set that scores negation and temporal failures separately from topical recall?
  • Build query slices by failure type, with at least one labelled supporting passage per query, and report recall@k per slice rather than one aggregate.
  • For negation, write pairs that differ only by the negating token and assert the retriever ranks the correct one above its twin.
  • For temporal queries, label the effective date and score whether the top-k contains the version valid at the query date, not just any version.
  • Keep the slices disjoint from tuning data and refresh them when the corpus changes.
If you fine-tune the embedder on negation pairs, what does that cost operationally and how do you stop it regressing on ordinary queries?
  • The cost is labelled contrastive pairs, a training run, and a full re-embed of the corpus because the new model's vectors are not comparable with the old ones.
  • Guard against regression with a broad retrieval suite covering paraphrase, identifiers and multi-constraint queries, plus per-slice recall before and after.
  • Use hard negatives that look relevant but are not, since easy negatives teach topical clustering rather than the negation distinction you are trying to add.
Which of these failures would you fix with BM25, which with a reranker, and which needs metadata filters or query decomposition?
  • BM25 is the fix for identifiers, product codes and exact numbers, because it matches tokens rather than meaning.
  • A reranker helps when the distinguishing contrast is present in the passage, such as a qualifier or a specific clause, because it reads query and candidate together.
  • A reranker handles negation that appears in the candidate text, so route those cases there first; temporal validity is different, because nothing in either passage says which version is current, so it needs an effective-date filter or a recency-ranked query.

Red flags

  • claims embeddings handle negation
  • reaches for a bigger model

Concepts this tests

Tags

Source: adilshamim8.medium.com

More in Embeddings & retrieval