Skip to content
AI Engineer Guide.
seniorReportedcurated

Why does swapping the embedding model force a full re-index, and how do you roll that out without downtime?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How do you keep deletes and updates consistent while the backfill is still running?
  • Write through a single ingestion path that applies the same upsert and delete to both collections, keyed by a deterministic chunk id, so replay is idempotent.
  • Track a per-document version or updated_at watermark and re-copy anything that changed after its backfill page was read.
  • Reconcile document ids and counts between source, old index and new index before the flip, and treat a non-zero miss rate as a blocker.
What rollback trigger would make you abort the migration, and how do you set it from evidence?
  • Abort if recall@10 or nDCG@10 on the frozen golden set falls outside the noise band measured on the old collection, or if the empty-result rate rises at all.
  • Measure the band by re-running the old model on the same set and against a rebuilt index several times, then set the trigger above that spread and below the regression you would ship.
  • Add an operational trigger for backfill lag or dual-write failure, because those corrupt the comparison even when offline scores look fine.
How do you handle a vendor deprecating the old model on 30 days' notice mid-migration?
  • Freeze a copy of the old model or its outputs where the licence allows, so rollback remains possible after the vendor deadline.
  • Compress the plan to the critical path: backfill first, validate on the golden set, and accept a shorter dual-write window rather than skipping validation.
  • Negotiate an extension in parallel and record the deprecation date in the index metadata, since the next migration will hit the same constraint.

Red flags

  • mixes vectors from two models
  • no rollback plan

Concepts this tests

Tags

Source: www.mockingly.ai

More in Embeddings & retrieval