seniorReportedcurated
You must choose an embedding model for a 900k-chunk support corpus. Walk me through the bake-off and the constraints that don't appear on a leaderboard.
30-second answer
- Freeze the corpus, chunking and golden set, vary only the embedding model, and score recall@5, recall@10, nDCG@10 and MRR on the same queries.
- Add the constraints a leaderboard omits: embedding latency, index memory, cost per million tokens, maximum sequence length against your chunk sizes, and required query and passage prefixes.
- Dropping an instruction prefix such as query: or passage: quietly costs several recall points, so verify the model's contract before comparing scores.
- Use MTEB only to shortlist about five candidates; the decision comes from your own labelled support queries, plus multilingual coverage, licence and whether self-hosting is required.
- A 3,072-dimension model may add two points of recall for double the RAM and slower search, and leaderboard rank does not transfer, so pick the smallest model that clears your own tolerance.
Likely follow-ups
Open a question to rehearse the answer.
A new model wins your offline bake-off by three nDCG points. What would stop you shipping it?
- Re-embedding 900k chunks and the storage increase may exceed the budget, so the win has to survive the cost of the migration.
- Check per-slice regressions: a gain on aggregate nDCG can hide losses on identifiers, short queries or a language you support.
- Verify operational constraints the benchmark ignores, such as sequence length, prefix requirements, licence and vendor stability.
- Confirm the gain is larger than run-to-run noise on the golden set before committing to a second collection.
How would you detect that a hosted embedding vendor silently changed the model behind a stable name?
- Record the model identifier and a hash of its configuration with every index build, and assert one model version per collection.
- Watch the score distribution, not only the top score: a shift in median similarity across queries is an early signal of an encoder change.
- Run a fixed golden set on a schedule and compare recall@k and MRR against the pinned baseline, because quality moves before an error appears.
How do you run this bake-off at 900k chunks without evaluating on a sample that biases the result?
- Sample by query type and document type rather than uniformly, so identifier queries and rare categories are represented in proportion to their importance.
- Hold out queries and their supporting passages before any tuning, and never inspect the held-out split while choosing a model.
- Deduplicate near-identical chunks in the sample, since duplicated passages inflate recall and make a model look better than it is.
Red flags
- picks by MTEB/popularity
- no recall measurement on own data
Concepts this tests
Tags
Source: ishwarj.com
More in Embeddings & retrieval
- Where do embeddings systematically fail as a retrieval mechanism, and which failures can you fix cheaply?
- You have 25M chunks at 1,536 dimensions. Size the index and tell me how you make it fit — then what changes at 10x.
- Why does swapping the embedding model force a full re-index, and how do you roll that out without downtime?