midReportedcurated
Where do embeddings systematically fail as a retrieval mechanism, and which failures can you fix cheaply?
30-second answer
- An embedding compresses a passage into one vector trained for topical similarity, so it captures what the passage is about rather than whether it entails the query.
- The systematic failures are negation, numbers and units, rare identifiers such as error codes, and temporal qualifiers, because those signals are weakly encoded or absent from the vector.
- A chunk saying no refunds after 30 days and one saying refunds after 30 days land close together, which is why a similarity threshold cannot separate them.
- BM25 recovers exact-token cases because inverse document frequency makes rare terms decisive, and a cross-encoder helps where the contrast is visible in the passage text.
- A cross-encoder can separate a negated candidate from its affirmative twin when the negation is in the retrieved text; what stays structural is a negation expressed only in the query, a conflict between retrieved documents, and temporal validity, which needs effective-date filters or a claim-level check.
Likely follow-ups
Open a question to rehearse the answer.
How would you build a golden set that scores negation and temporal failures separately from topical recall?
- Build query slices by failure type, with at least one labelled supporting passage per query, and report recall@k per slice rather than one aggregate.
- For negation, write pairs that differ only by the negating token and assert the retriever ranks the correct one above its twin.
- For temporal queries, label the effective date and score whether the top-k contains the version valid at the query date, not just any version.
- Keep the slices disjoint from tuning data and refresh them when the corpus changes.
If you fine-tune the embedder on negation pairs, what does that cost operationally and how do you stop it regressing on ordinary queries?
- The cost is labelled contrastive pairs, a training run, and a full re-embed of the corpus because the new model's vectors are not comparable with the old ones.
- Guard against regression with a broad retrieval suite covering paraphrase, identifiers and multi-constraint queries, plus per-slice recall before and after.
- Use hard negatives that look relevant but are not, since easy negatives teach topical clustering rather than the negation distinction you are trying to add.
Which of these failures would you fix with BM25, which with a reranker, and which needs metadata filters or query decomposition?
- BM25 is the fix for identifiers, product codes and exact numbers, because it matches tokens rather than meaning.
- A reranker helps when the distinguishing contrast is present in the passage, such as a qualifier or a specific clause, because it reads query and candidate together.
- A reranker handles negation that appears in the candidate text, so route those cases there first; temporal validity is different, because nothing in either passage says which version is current, so it needs an effective-date filter or a recency-ranked query.
Red flags
- claims embeddings handle negation
- reaches for a bigger model
Concepts this tests
Tags
Source: adilshamim8.medium.com
More in Embeddings & retrieval
- You must choose an embedding model for a 900k-chunk support corpus. Walk me through the bake-off and the constraints that don't appear on a leaderboard.
- You have 25M chunks at 1,536 dimensions. Size the index and tell me how you make it fit — then what changes at 10x.
- Why does swapping the embedding model force a full re-index, and how do you roll that out without downtime?