Skip to content
AI Engineer Guide.
seniorReportedcurated

You must choose an embedding model for a 900k-chunk support corpus. Walk me through the bake-off and the constraints that don't appear on a leaderboard.

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

A new model wins your offline bake-off by three nDCG points. What would stop you shipping it?
  • Re-embedding 900k chunks and the storage increase may exceed the budget, so the win has to survive the cost of the migration.
  • Check per-slice regressions: a gain on aggregate nDCG can hide losses on identifiers, short queries or a language you support.
  • Verify operational constraints the benchmark ignores, such as sequence length, prefix requirements, licence and vendor stability.
  • Confirm the gain is larger than run-to-run noise on the golden set before committing to a second collection.
How would you detect that a hosted embedding vendor silently changed the model behind a stable name?
  • Record the model identifier and a hash of its configuration with every index build, and assert one model version per collection.
  • Watch the score distribution, not only the top score: a shift in median similarity across queries is an early signal of an encoder change.
  • Run a fixed golden set on a schedule and compare recall@k and MRR against the pinned baseline, because quality moves before an error appears.
How do you run this bake-off at 900k chunks without evaluating on a sample that biases the result?
  • Sample by query type and document type rather than uniformly, so identifier queries and rare categories are represented in proportion to their importance.
  • Hold out queries and their supporting passages before any tuning, and never inspect the held-out split while choosing a model.
  • Deduplicate near-identical chunks in the sample, since duplicated passages inflate recall and make a model look better than it is.

Red flags

  • picks by MTEB/popularity
  • no recall measurement on own data

Concepts this tests

Tags

Source: ishwarj.com

More in Embeddings & retrieval