Why it matters
Retrieval quality is capped by what ingestion actually stored. A parser that drops tables, a loader that stops at the first page of results, or a retry that duplicates a document all look like retrieval bugs later. The pipeline therefore needs the properties of any data pipeline — idempotency, provenance, dead-letter handling — plus one of its own: the chunk is the unit of truth, so its identity must survive re-parsing and re-embedding.
Mental model
Ingestion is a loading dock: goods arrive in mixed packaging, get unpacked, labelled and shelved, and anything damaged goes to a quarantine bay rather than the shelf. The metaphor breaks because damaged documents can still be partially useful, so the decision is not accept or reject but which parts to trust.
How it works
Synthetic ingestion record · run 09:14Z
One document · three chunks · one batch table
bounding boxes
Step 1A stable identity
The document id comes from the source; the content hash catches the same bytes under a new id. Chunk ids are derived from both, so a re-parse can replace them instead of adding a second copy.
INGEST RUN · CONTRACTS · 2026-11-17T09:14Z
Ingestion record — Northwind MSA
| Document | Hash | State | Error |
|---|---|---|---|
| contracts:8841 | 9f2c…4a1 | ✓indexed | — |
| contracts:8842 | 9f2c…4a1 | ≡duplicate | same hash |
| contracts:8843 | 71ab…0c3 | Qquarantined | ✕parse_failed |
| contracts:8844 | 38de…991 | ✓indexed | — |
| contracts:8845 | 5b07…22e | ✓indexed | — |
cursor 09:14:12Z · 412 documents · 3,806 chunks · 0 skipped without a record
Step record
- Identity
- source id + hash
- Chunk ids
- docid:ordinal
- Reuse
- safe on retry
The text version summarises this figure; select a region for its full text.
Figure. One document's ingestion record: a source id and content hash, three deterministic chunks, and a batch table where a duplicate hash, a quarantined file and an indexed document are separate states rather than silent losses.
What to notice. Synthetic batch. Hashes, counts and the watermark are illustrative. The figure shows identity, atomicity and reconciliation state, not measured throughput or a real dedupe rate.
Text version
An ingestion record for a supplier agreement. Four fields hold the document id contracts:8841, revision r7, a content hash and a count of three chunks. Three chunk regions show the parties and term, liability and data-processing sections. A batch table lists five documents with their hashes and states: three indexed, one duplicate hash under a new id, and one quarantined file. A write record says three chunks and three vectors committed together. A footer holds the cursor, 412 documents and 3,806 chunks. Chunk boundaries and chunk ids appear as an overlay; selecting a chunk shows its text and its chunk label.
- Assign each source document a stable identity from its native id plus a content hash, and derive chunk ids from that identity.
- Load incrementally with a cursor or change token, recording the watermark so a restart resumes rather than rescans.
- Parse and normalise into a common structure with page or section offsets preserved.
- Chunk deterministically from the parsed structure so the same input yields the same chunk boundaries.
- Write chunks and vectors in one transaction per document, or mark the document incomplete until all chunks land.
- Quarantine documents that fail parsing or exceed limits, with the error and a retry path, instead of dropping them.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Full re-ingest per change | Simple, deterministic, no drift | Cost and time proportional to corpus size | The corpus is large and changes hourly |
| Incremental by cursor | Cheap and frequent updates | Missed changes when the source lacks a reliable cursor | The source API only exposes modification times loosely |
| Fail the batch on error | No partial state | One bad file blocks thousands of good ones | Sources contain heterogeneous formats |
| Quarantine and continue | Throughput and visibility | A growing backlog nobody owns | There is no alerting on quarantine size |
Good for
Simple, deterministic, no drift
Cost
Cost and time proportional to corpus size
Fails when
The corpus is large and changes hourly
- The corpus is large and changes hourly
- The source API only exposes modification times loosely
- Sources contain heterogeneous formats
- There is no alerting on quarantine size
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Full re-ingest per change is good for Simple, deterministic, no drift, costs Cost and time proportional to corpus size, and fails when The corpus is large and changes hourly. Incremental by cursor is good for Cheap and frequent updates, costs Missed changes when the source lacks a reliable cursor, and fails when The source API only exposes modification times loosely. Fail the batch on error is good for No partial state, costs One bad file blocks thousands of good ones, and fails when Sources contain heterogeneous formats. Quarantine and continue is good for Throughput and visibility, costs A growing backlog nobody owns, and fails when There is no alerting on quarantine size
Failure modes
- Duplicate documents after retry. The same file is indexed twice under different ids and both appear in results. Debug: group the index by content hash and list hashes with more than one id.
- Cursor reset on deploy. The loader restarts from the beginning and either duplicates work or, worse, reindexes an old snapshot. Debug: log the watermark at start and end of each run and confirm it only moves forward.
- Partial document write. Half the chunks are indexed when the job dies, so the document is retrievable but incomplete. Debug: compare chunk counts per document with the expected count from the parser.
In production
Monitor documents in, chunks out, quarantine size and ingestion lag per source. Cap retries per document and alert on a watermark that has not advanced within its expected interval.
Common misconceptions
People think: Ingestion is a one-off backfill.
Actually: It runs continuously, and its incremental path is the one that breaks in production.
People think: A parse failure is a data problem, not an engineering one.
Actually: Unhandled parse failures become silent retrieval gaps that no amount of model tuning can fix.
Interview angle
Cover document identity, idempotency and quarantine, then connect ingestion lag to answer freshness. Name the partial-write failure and how you would detect it. A script that reads files and calls an embedding API, with no story for retries, deletes or provenance, is a demo rather than an ingestion design.
Sources
- Apache Tika — format detection and text extraction across heterogeneous documents.
- Unstructured documentation — partitioning documents into typed elements for downstream chunking.
- W3C, PROV-O: The PROV Ontology — the vocabulary for recording provenance of derived data.