Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 25 of 33

Whiteboard: ingestion

0 of 33 complete

Why it matters

Ingestion is where most retrieval and analytics problems originate, and it is the part candidates sketch least carefully. A whiteboard answer that jumps to a vector database skips the questions that decide whether the corpus is correct: what identifies a document, what happens when it is updated, how a deleted document is removed, and how a re-index is performed without serving stale and new content together. Those are the questions the interviewer is listening for.

Mental model

Ingestion is a mailroom: receive, open, sort, label, file, and remove what is outdated. The metaphor breaks because mail arrives once, while sources resend the same document with small edits and no clear signal about which version is current, so the mailroom needs a rule for the second copy rather than only the first.

How it works

Mechanism, step by stepInteractive
  1. 1List each source, its access method, its delivery guarantee and its update and delete semantics.
  2. 2Define a stable document id and a content hash, so duplicates and changes are distinguishable.
  3. 3Describe extraction and normalisation, including what happens to tables, images and encodings.
  4. 4Choose a staging area and a write pattern that is idempotent per document version.
  5. 5Explain how a re-index runs alongside the live index and when traffic switches.
  6. 6Name one failure, its detection check and the alert that would catch it.

Step 1 of 6

List each source, its access method, its delivery guarantee and its update and delete semantics.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. List each source, its access method, its delivery guarantee and its update and delete sem… 2. Define a stable document id and a content hash, so duplicates and changes are distinguish… 3. Describe extraction and normalisation, including what happens to tables, images and encod… 4. Choose a staging area and a write pattern that is idempotent per document version. 5. Explain how a re-index runs alongside the live index and when traffic switches. 6. Name one failure, its detection check and the alert that would catch it.

  1. List each source, its access method, its delivery guarantee and its update and delete semantics.
  2. Define a stable document id and a content hash, so duplicates and changes are distinguishable.
  3. Describe extraction and normalisation, including what happens to tables, images and encodings.
  4. Choose a staging area and a write pattern that is idempotent per document version.
  5. Explain how a re-index runs alongside the live index and when traffic switches.
  6. Name one failure, its detection check and the alert that would catch it.

Trade-offs

ChoiceGood forCostFails when
Full re-index per syncSimple and consistentExpensive and slow at corpus scaleThe corpus is large and changes hourly
Incremental upsertCheap and fastDeletion and drift are easy to missSources do not emit delete events
Dual-write to two indexesZero-downtime migrationTwo write paths to keep consistentOne write fails silently
Pull on a scheduleNo source changes neededLatency and API rate limitsThe source has no incremental endpoint
Trade-off mapInteractive

Good for

Simple and consistent

Cost

Expensive and slow at corpus scale

Fails when

The corpus is large and changes hourly

  1. The corpus is large and changes hourly
  2. Sources do not emit delete events
  3. One write fails silently
  4. The source has no incremental endpoint

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full re-index per sync is good for Simple and consistent, costs Expensive and slow at corpus scale, and fails when The corpus is large and changes hourly. Incremental upsert is good for Cheap and fast, costs Deletion and drift are easy to miss, and fails when Sources do not emit delete events. Dual-write to two indexes is good for Zero-downtime migration, costs Two write paths to keep consistent, and fails when One write fails silently. Pull on a schedule is good for No source changes needed, costs Latency and API rate limits, and fails when The source has no incremental endpoint

Failure modes

  • Ghost documents. Deleted source documents remain retrievable and cite removed content. Debug: anti-join index ids against the current source listing and count orphans.
  • Duplicate chunks. A re-sync inserts the same document again and retrieval returns near-identical passages. Debug: group by content hash and count rows greater than one.
  • Mixed index versions. A partial re-index serves old and new chunking together. Debug: compare the chunker version recorded per vector against the expected version.

In production

Track document counts per source, orphan counts and index version per vector. Run the anti-join and the hash-duplicate check on a schedule, and rehearse a full re-index before you need one.

Common misconceptions

People think: Ingestion is a one-off script.

Actually: Sources change, delete and resend, so ingestion is a recurring pipeline with its own failure modes.

People think: A vector database handles deduplication.

Actually: It stores what you write; identity and versioning are decisions made before the write.

Interview angle

Strong answers cover source contracts, identity, idempotent writes, deletion and re-index, then name a failure and its check. Weak answers start at the embedding model and never mention updates or deletes.

Sources