Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 18 of 57

Ingestion

0 of 57 complete

Why it matters

Retrieval quality is capped by what ingestion actually stored. A parser that drops tables, a loader that stops at the first page of results, or a retry that duplicates a document all look like retrieval bugs later. The pipeline therefore needs the properties of any data pipeline — idempotency, provenance, dead-letter handling — plus one of its own: the chunk is the unit of truth, so its identity must survive re-parsing and re-embedding.

Mental model

Ingestion is a loading dock: goods arrive in mixed packaging, get unpacked, labelled and shelved, and anything damaged goes to a quarantine bay rather than the shelf. The metaphor breaks because damaged documents can still be partially useful, so the decision is not accept or reject but which parts to trust.

How it works

One document, from identity to chunksInteractive

Synthetic ingestion record · run 09:14Z

One document · three chunks · one batch table

bounding boxes

Step 1/6 · A stable identity

Step 1A stable identity

The document id comes from the source; the content hash catches the same bytes under a new id. Chunk ids are derived from both, so a re-parse can replace them instead of adding a second copy.

INGEST RUN · CONTRACTS · 2026-11-17T09:14Z

, running header, region 1. Select to inspect.

Ingestion record — Northwind MSA

, title, region 2. Select to inspect.
Document idcontracts:8841
, form field Document id, region 3. Select to inspect.
Revisionr7
, form field Revision, region 4. Select to inspect.
Content hash9f2c…4a1
, form field Content hash, region 5. Select to inspect.
Chunks3
, form field Chunks, region 6. Select to inspect.
Text

1 · Parties and term — This agreement starts on 1 October 2026 and renews annually unless either party gives 60 days' notice.

, paragraph, region 7. Select to inspect.
Text

2 · Liability — Each party's aggregate liability is capped at the fees paid in the twelve months before the claim.

, paragraph, region 8. Select to inspect.
Text

3 · Data processing — Processor handles personal data only on documented instructions and deletes it on termination.

, paragraph, region 9. Select to inspect.
BATCH · 5 DOCUMENTS
DocumentHashStateError
contracts:88419f2c…4a1✓indexed—
contracts:88429f2c…4a1≡duplicatesame hash
contracts:884371ab…0c3Qquarantined✕parse_failed
contracts:884438de…991✓indexed—
contracts:88455b07…22e✓indexed—
, table BATCH · 5 DOCUMENTS, region 10. Select to inspect.
Write3 chunks + 3 vectors · committed together
, form field Write, region 11. Select to inspect.

cursor 09:14:12Z · 412 documents · 3,806 chunks · 0 skipped without a record

, running footer, region 12. Select to inspect.
12 regions

Step record

Identity
source id + hash
Chunk ids
docid:ordinal
Reuse
safe on retry

The text version summarises this figure; select a region for its full text.

Markspassduplicate contentquarantinedcheck failed

Figure. One document's ingestion record: a source id and content hash, three deterministic chunks, and a batch table where a duplicate hash, a quarantined file and an indexed document are separate states rather than silent losses.

What to notice. Synthetic batch. Hashes, counts and the watermark are illustrative. The figure shows identity, atomicity and reconciliation state, not measured throughput or a real dedupe rate.

Text version

An ingestion record for a supplier agreement. Four fields hold the document id contracts:8841, revision r7, a content hash and a count of three chunks. Three chunk regions show the parties and term, liability and data-processing sections. A batch table lists five documents with their hashes and states: three indexed, one duplicate hash under a new id, and one quarantined file. A write record says three chunks and three vectors committed together. A footer holds the cursor, 412 documents and 3,806 chunks. Chunk boundaries and chunk ids appear as an overlay; selecting a chunk shows its text and its chunk label.

  1. Assign each source document a stable identity from its native id plus a content hash, and derive chunk ids from that identity.
  2. Load incrementally with a cursor or change token, recording the watermark so a restart resumes rather than rescans.
  3. Parse and normalise into a common structure with page or section offsets preserved.
  4. Chunk deterministically from the parsed structure so the same input yields the same chunk boundaries.
  5. Write chunks and vectors in one transaction per document, or mark the document incomplete until all chunks land.
  6. Quarantine documents that fail parsing or exceed limits, with the error and a retry path, instead of dropping them.

Trade-offs

ChoiceGood forCostFails when
Full re-ingest per changeSimple, deterministic, no driftCost and time proportional to corpus sizeThe corpus is large and changes hourly
Incremental by cursorCheap and frequent updatesMissed changes when the source lacks a reliable cursorThe source API only exposes modification times loosely
Fail the batch on errorNo partial stateOne bad file blocks thousands of good onesSources contain heterogeneous formats
Quarantine and continueThroughput and visibilityA growing backlog nobody ownsThere is no alerting on quarantine size
Trade-off mapInteractive

Good for

Simple, deterministic, no drift

Cost

Cost and time proportional to corpus size

Fails when

The corpus is large and changes hourly

  1. The corpus is large and changes hourly
  2. The source API only exposes modification times loosely
  3. Sources contain heterogeneous formats
  4. There is no alerting on quarantine size

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Full re-ingest per change is good for Simple, deterministic, no drift, costs Cost and time proportional to corpus size, and fails when The corpus is large and changes hourly. Incremental by cursor is good for Cheap and frequent updates, costs Missed changes when the source lacks a reliable cursor, and fails when The source API only exposes modification times loosely. Fail the batch on error is good for No partial state, costs One bad file blocks thousands of good ones, and fails when Sources contain heterogeneous formats. Quarantine and continue is good for Throughput and visibility, costs A growing backlog nobody owns, and fails when There is no alerting on quarantine size

Failure modes

  • Duplicate documents after retry. The same file is indexed twice under different ids and both appear in results. Debug: group the index by content hash and list hashes with more than one id.
  • Cursor reset on deploy. The loader restarts from the beginning and either duplicates work or, worse, reindexes an old snapshot. Debug: log the watermark at start and end of each run and confirm it only moves forward.
  • Partial document write. Half the chunks are indexed when the job dies, so the document is retrievable but incomplete. Debug: compare chunk counts per document with the expected count from the parser.

In production

Monitor documents in, chunks out, quarantine size and ingestion lag per source. Cap retries per document and alert on a watermark that has not advanced within its expected interval.

Common misconceptions

People think: Ingestion is a one-off backfill.

Actually: It runs continuously, and its incremental path is the one that breaks in production.

People think: A parse failure is a data problem, not an engineering one.

Actually: Unhandled parse failures become silent retrieval gaps that no amount of model tuning can fix.

Interview angle

Cover document identity, idempotency and quarantine, then connect ingestion lag to answer freshness. Name the partial-write failure and how you would detect it. A script that reads files and calls an embedding API, with no story for retries, deletes or provenance, is a demo rather than an ingestion design.

Sources