Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 28 of 57

Indexing

0 of 57 complete

Why it matters

Retrieval only sees what the index contains, in the form the index stored it. Building that structure is a batch job with its own failure modes: partial writes, mismatched analyzers, vectors from a different model, documents indexed twice. Treating the index as disposable and reproducible is what makes those failures recoverable, because the fix is a rebuild rather than a repair.

Mental model

An index is a printed catalogue derived from a library. If a card is wrong you reprint the catalogue; you do not edit the card in place and hope. The metaphor breaks because reprinting is cheap for a catalogue and expensive for millions of vectors, so incremental repair exists for cost reasons.

How it works

Two index builds on one disk budgetInteractive

Index budget · GB

40 of 100 GB used

Budget lines · GB

  • v0 bytes40 GB
  • v1 bytes0 GB
  • v2 bytes0 GB

Step · 1 of 9

v0 serves the whole budget it needs

The live index occupies 40 GB of a 100 GB budget and every query reaches it through one alias. Sixty GB is free; nothing is being built and the old index is the only copy.

Next: v1 builds into a fresh index

Alias
v0
Resident builds
1
Validation
—
Free (GB)
60
Step 1/9 · v0 serves the whole budget it needs

Figure. A 100 GB index budget. The live index keeps serving through an alias while a new build occupies its own space, so both copies are resident during the build and the old index survives a failed validation. A failed 40 GB build is deleted and rebuilt; the alias moves only after validation. Sizes are synthetic; query latency during the swap is not modelled.

What to notice. The alias is the cutover, not the build. A build that never passes validation (step 3) is deleted and the live index is untouched, because nothing wrote to v0. Step 6 is the first cutover, and v0 stays resident for a rollback window, so the cost of the swap is double storage rather than downtime. Steps 8 and 9 repeat the shape for an analyzer change: the mapping is baked into the stored terms, so the only way to change it is a fresh build and another swap.

Text version

A 100 GB stacked budget against a 100 GB capacity marker. Three segments: v0, v1 and v2. Step 1: v0 40 GB, alias v0, 60 free. Step 2: v1 build 20 GB, v0 40, 40 free. Step 3: v1 reaches 40 GB but validation fails, total 80, alias still v0. Step 4: v1 deleted, total 40, 60 free. Step 5: v1 rebuilt to 40 GB alongside v0, total 80, 20 free. Step 6: validation passes, alias v1, v0 retained, total 80. Step 7: v0 deleted, total 40, alias v1. Step 8: v2 builds 40 GB while v1 serves, total 80. Step 9: v2 validates and takes the alias, v1 retained, total 80. Counters show alias v0, v0, v0, v0, v0, v1, v1, v1, v2; resident builds 1, 2, 2, 1, 2, 2, 1, 2, 2; validation fail at step 3 and pass at steps 6 and 9; free 60, 40, 20, 60, 20, 20, 60, 20, 20 GB.

  1. Define the index mapping explicitly: text fields with their analyzers, vector fields with dimension and metric, metadata with types.
  2. Build into a fresh index rather than mutating the live one, so a failed build leaves the old index serving.
  3. Write documents in batches with deterministic ids derived from source identity and chunk position.
  4. Validate after the build: document count, vector count, dimension, and a sample query against known answers.
  5. Swap an alias or pointer to the new index only after validation passes, and keep the old one for rollback.
  6. Record the pipeline configuration (chunker, model, analyzer, prompt) alongside the index version.

Trade-offs

ChoiceGood forCostFails when
Rebuild into a new indexAtomic swap and easy rollbackDouble storage during the buildThe corpus is large and storage is capped
In-place updatesNo duplicate storagePartial states are visible to queriesSchema or model changes are involved
Alias swapZero-downtime cutoverRequires discipline about which name queries useClients hard-code a physical index name
Many small indexesIsolation per tenant or languageOperational overhead per indexTenants are numerous and small
Trade-off mapInteractive

Good for

Atomic swap and easy rollback

Cost

Double storage during the build

Fails when

The corpus is large and storage is capped

  1. The corpus is large and storage is capped
  2. Schema or model changes are involved
  3. Clients hard-code a physical index name
  4. Tenants are numerous and small

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Rebuild into a new index is good for Atomic swap and easy rollback, costs Double storage during the build, and fails when The corpus is large and storage is capped. In-place updates is good for No duplicate storage, costs Partial states are visible to queries, and fails when Schema or model changes are involved. Alias swap is good for Zero-downtime cutover, costs Requires discipline about which name queries use, and fails when Clients hard-code a physical index name. Many small indexes is good for Isolation per tenant or language, costs Operational overhead per index, and fails when Tenants are numerous and small

Failure modes

  • Partial build promoted. The alias points at an index missing half the corpus and recall drops without an error. Debug: compare document counts between source and index before every swap.
  • Mapping change without rebuild. A new analyzer is applied to queries but not to stored terms, so matching silently changes. Debug: run the analyzer on a stored document's field and compare with the query-side output.
  • Vector dimension mismatch. A model change writes vectors the index cannot compare, and the write either fails or corrupts distances. Debug: assert dimension and model id on every write batch.

In production

Record build duration, document counts and validation results per index version. Keep the previous index warm for a rollback window, and alert if the live alias points at an index older than its refresh interval.

Common misconceptions

People think: The index is the source of truth.

Actually: It is derived; losing it costs a rebuild, not data, provided the source and pipeline config are intact.

People think: Reindexing is a background chore.

Actually: It is a deployment with a cutover, a validation gate and a rollback path.

Interview angle

Describe the mapping, the build-validate-swap sequence, and the configuration recorded for reproducibility. Treat reindexing as a deployment with a rollback path. Running an embedding job and inserting rows, with no validation or cutover plan, is the answer that ends in a partial index.

Sources