Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 19 of 57

Loaders

0 of 57 complete

Why it matters

Most retrieval failures that look like model problems start with a loader that never saw the document. Connectors paginate, throttle and change shape between sources, and each one has a different notion of modification time and deletion. The loader's real output is not text but a record: bytes plus identity plus timestamps plus access metadata, sufficient for the rest of the pipeline to be idempotent.

Mental model

A loader is a courier service between a warehouse and your shelf. It must know which parcels it already delivered, which were returned and which warehouse door to use. The metaphor breaks because the warehouse keeps editing parcels in place, so delivery is a continuous reconciliation rather than a one-time shipment.

How it works

A loader's output is a recordInteractive

Synthetic source record · legal contracts

Listing page 1 of 13 · one document opened

bounding boxes

Step 1/6 · List to exhaustion

Step 1List to exhaustion

The listing API returns pages of items, not the folder. Follow the cursor to exhaustion and log the count; an early stop looks like a smaller corpus, not an error.

SHARED DRIVE · LEGAL / CONTRACTS · SYNC 09:14Z

, running header, region 1. Select to inspect.

Supplier agreement — Northwind

, title, region 2. Select to inspect.
Text

Revision r7 · 42 pages · 1.2 MB

, paragraph, region 3. Select to inspect.
Source iddrive:legal/8841
, form field Source id, region 4. Select to inspect.
Revisionr7 · 2026-11-16
, form field Revision, region 5. Select to inspect.
Modified2026-11-16 14:02Z
, form field Modified, region 6. Select to inspect.
Content hashsha256 9f2c…4a1
, form field Content hash, region 7. Select to inspect.
PermissionsLegal · 42 principals
, form field Permissions, region 8. Select to inspect.
FormatDOCX (native)
, form field Format, region 9. Select to inspect.

Clause 4 — limitation of liability

, heading, region 10. Select to inspect.
Text

Each party's aggregate liability is capped at the fees paid in the twelve months before the claim. The cap does not apply to confidentiality breaches or indemnities. This clause survives termination.

, paragraph, region 11. Select to inspect.
FOLDER LISTING · 1,284 ITEMS · SHOWING 8
NameModifiedState
MSA — Northwind.docx11-16 14:02~changed
NDA — Contoso.pdf11-14 09:40+new
SOW — Fabrikam.docx11-02 11:15−deleted
DPA — Northwind.pdf10-28 16:31→moved
Vendor list.xlsx10-21 13:08~changed
Insurance — 2026.pdf09-30 08:12+new
Old SOW — 2024.docx2024-06-01−deleted
Archive / 20252025-12-19=unchanged
, table FOLDER LISTING · 1,284 ITEMS · SHOWING 8, region 12. Select to inspect.
Sync summary1,284 listed · 1,284 fetched · 0 failed
, form field Sync summary, region 13. Select to inspect.

cursor 2026-11-17T09:14Z · next sync reads only items after this watermark

, running footer, region 14. Select to inspect.
14 regions

Step record

Pages read
13 / 13
Items listed
1,284
Items fetched
1,284

The text version summarises this figure; select a region for its full text.

Markschanged documentnew documentdeleted documentmoved documentunchanged

Figure. A loader's output is a record: source id, revision, content hash, native format and permissions travel with the bytes. The listing shows what a sync must reconcile — new, changed, deleted and moved items — and the watermark that makes the next run incremental.

What to notice. Synthetic shared drive and listing. Counts, hashes and timestamps are illustrative. The figure shows the metadata a loader must keep and the listing states it must reconcile, not a connector's measured throughput or a real permission model. Throttling and backoff are not shown.

Text version

A synthetic document record for a supplier agreement on a shared legal drive. Six form fields hold the source id, revision, modified time, content hash, permissions and received format, which is native DOCX. A folder listing with eight item rows shows new, changed, deleted, moved and unchanged items, and a sync summary reports that 1,284 items were listed and fetched with no failures. Steps light the listing, the format field, the identity fields, the permission field, the move and delete rows, and then the watermark in the footer. Selecting any region shows its value and why the pipeline needs it.

  1. Enumerate the source with its native listing API, following pagination to exhaustion and logging page counts.
  2. Fetch bytes in the richest available format rather than a lossy export, and record the format actually received.
  3. Store the source id, revision or modified time, and a content hash beside the document for deduplication.
  4. Capture permission metadata at load time so access filtering has something to filter on.
  5. Handle throttling with backoff and a per-source concurrency cap rather than parallelising without limit.
  6. Record deletions and moves explicitly, since a document that disappears from the listing must disappear from the index.

Trade-offs

ChoiceGood forCostFails when
Native format fetchPreserves structure for parsingLarger payloads and slower syncThe source rate-limits bulk downloads
Export to plain textSimple and fastLoses headings, tables and layoutThe corpus is reports or spreadsheets
Full listing each syncCatches every change including deletionsCost grows with corpus sizeThe source has millions of files
Delta by modified timeCheap and frequentMisses deletions and clock skewDocuments are removed without a tombstone
Trade-off mapInteractive

Good for

Preserves structure for parsing

Cost

Larger payloads and slower sync

Fails when

The source rate-limits bulk downloads

  1. The source rate-limits bulk downloads
  2. The corpus is reports or spreadsheets
  3. The source has millions of files
  4. Documents are removed without a tombstone

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Native format fetch is good for Preserves structure for parsing, costs Larger payloads and slower sync, and fails when The source rate-limits bulk downloads. Export to plain text is good for Simple and fast, costs Loses headings, tables and layout, and fails when The corpus is reports or spreadsheets. Full listing each sync is good for Catches every change including deletions, costs Cost grows with corpus size, and fails when The source has millions of files. Delta by modified time is good for Cheap and frequent, costs Misses deletions and clock skew, and fails when Documents are removed without a tombstone

Failure modes

  • Pagination stopped early. Only the first page of a folder is indexed and nobody notices until a user asks about a missing document. Debug: compare the loader's item count with the source's reported total.
  • Lossy export path. Tables arrive as run-together text and chunking produces unusable fragments. Debug: inspect the raw loaded text for one document known to contain a table.
  • Missing permission metadata. Access filtering has no field to filter on, so documents are either over-exposed or hidden from everyone. Debug: count loaded documents with an empty permission field.

In production

Emit per-source metrics: items listed, items fetched, bytes, failures and sync duration. Alert when fetched differs from listed, which is the earliest signal of a truncated crawl.

Common misconceptions

People think: A loader just downloads files.

Actually: It also establishes identity, freshness and permissions, which the rest of the pipeline depends on.

People think: Exporting everything to text simplifies the pipeline.

Actually: It simplifies loading and makes parsing and chunking worse, because structure is discarded before it can be used.

Interview angle

Treat the loader as a reconciliation job: pagination, identity, timestamps, deletions and permissions. Name the truncated-crawl failure and the metric that reveals it. Listing connector names without explaining how a re-sync avoids duplicates or catches deletions is the shallow answer.

Sources