Why it matters
Most retrieval failures that look like model problems start with a loader that never saw the document. Connectors paginate, throttle and change shape between sources, and each one has a different notion of modification time and deletion. The loader's real output is not text but a record: bytes plus identity plus timestamps plus access metadata, sufficient for the rest of the pipeline to be idempotent.
Mental model
A loader is a courier service between a warehouse and your shelf. It must know which parcels it already delivered, which were returned and which warehouse door to use. The metaphor breaks because the warehouse keeps editing parcels in place, so delivery is a continuous reconciliation rather than a one-time shipment.
How it works
Synthetic source record · legal contracts
Listing page 1 of 13 · one document opened
bounding boxes
Step 1List to exhaustion
The listing API returns pages of items, not the folder. Follow the cursor to exhaustion and log the count; an early stop looks like a smaller corpus, not an error.
SHARED DRIVE · LEGAL / CONTRACTS · SYNC 09:14Z
Supplier agreement — Northwind
Clause 4 — limitation of liability
| Name | Modified | State |
|---|---|---|
| MSA — Northwind.docx | 11-16 14:02 | ~changed |
| NDA — Contoso.pdf | 11-14 09:40 | +new |
| SOW — Fabrikam.docx | 11-02 11:15 | −deleted |
| DPA — Northwind.pdf | 10-28 16:31 | →moved |
| Vendor list.xlsx | 10-21 13:08 | ~changed |
| Insurance — 2026.pdf | 09-30 08:12 | +new |
| Old SOW — 2024.docx | 2024-06-01 | −deleted |
| Archive / 2025 | 2025-12-19 | =unchanged |
cursor 2026-11-17T09:14Z · next sync reads only items after this watermark
Step record
- Pages read
- 13 / 13
- Items listed
- 1,284
- Items fetched
- 1,284
The text version summarises this figure; select a region for its full text.
Figure. A loader's output is a record: source id, revision, content hash, native format and permissions travel with the bytes. The listing shows what a sync must reconcile — new, changed, deleted and moved items — and the watermark that makes the next run incremental.
What to notice. Synthetic shared drive and listing. Counts, hashes and timestamps are illustrative. The figure shows the metadata a loader must keep and the listing states it must reconcile, not a connector's measured throughput or a real permission model. Throttling and backoff are not shown.
Text version
A synthetic document record for a supplier agreement on a shared legal drive. Six form fields hold the source id, revision, modified time, content hash, permissions and received format, which is native DOCX. A folder listing with eight item rows shows new, changed, deleted, moved and unchanged items, and a sync summary reports that 1,284 items were listed and fetched with no failures. Steps light the listing, the format field, the identity fields, the permission field, the move and delete rows, and then the watermark in the footer. Selecting any region shows its value and why the pipeline needs it.
- Enumerate the source with its native listing API, following pagination to exhaustion and logging page counts.
- Fetch bytes in the richest available format rather than a lossy export, and record the format actually received.
- Store the source id, revision or modified time, and a content hash beside the document for deduplication.
- Capture permission metadata at load time so access filtering has something to filter on.
- Handle throttling with backoff and a per-source concurrency cap rather than parallelising without limit.
- Record deletions and moves explicitly, since a document that disappears from the listing must disappear from the index.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Native format fetch | Preserves structure for parsing | Larger payloads and slower sync | The source rate-limits bulk downloads |
| Export to plain text | Simple and fast | Loses headings, tables and layout | The corpus is reports or spreadsheets |
| Full listing each sync | Catches every change including deletions | Cost grows with corpus size | The source has millions of files |
| Delta by modified time | Cheap and frequent | Misses deletions and clock skew | Documents are removed without a tombstone |
Good for
Preserves structure for parsing
Cost
Larger payloads and slower sync
Fails when
The source rate-limits bulk downloads
- The source rate-limits bulk downloads
- The corpus is reports or spreadsheets
- The source has millions of files
- Documents are removed without a tombstone
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Native format fetch is good for Preserves structure for parsing, costs Larger payloads and slower sync, and fails when The source rate-limits bulk downloads. Export to plain text is good for Simple and fast, costs Loses headings, tables and layout, and fails when The corpus is reports or spreadsheets. Full listing each sync is good for Catches every change including deletions, costs Cost grows with corpus size, and fails when The source has millions of files. Delta by modified time is good for Cheap and frequent, costs Misses deletions and clock skew, and fails when Documents are removed without a tombstone
Failure modes
- Pagination stopped early. Only the first page of a folder is indexed and nobody notices until a user asks about a missing document. Debug: compare the loader's item count with the source's reported total.
- Lossy export path. Tables arrive as run-together text and chunking produces unusable fragments. Debug: inspect the raw loaded text for one document known to contain a table.
- Missing permission metadata. Access filtering has no field to filter on, so documents are either over-exposed or hidden from everyone. Debug: count loaded documents with an empty permission field.
In production
Emit per-source metrics: items listed, items fetched, bytes, failures and sync duration. Alert when fetched differs from listed, which is the earliest signal of a truncated crawl.
Common misconceptions
People think: A loader just downloads files.
Actually: It also establishes identity, freshness and permissions, which the rest of the pipeline depends on.
People think: Exporting everything to text simplifies the pipeline.
Actually: It simplifies loading and makes parsing and chunking worse, because structure is discarded before it can be used.
Interview angle
Treat the loader as a reconciliation job: pagination, identity, timestamps, deletions and permissions. Name the truncated-crawl failure and the metric that reveals it. Listing connector names without explaining how a re-sync avoids duplicates or catches deletions is the shallow answer.
Sources
- Apache Tika — detecting and extracting content from many source formats.
- Unstructured documentation — connectors and partitioning for document ingestion.
- RFC 9110, HTTP Semantics — conditional requests and caching semantics that incremental loaders rely on.