Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 22 of 57

Metadata enrichment

0 of 57 complete

Why it matters

Raw chunk text cannot answer questions about recency, ownership or document type, and it cannot be filtered without scanning. Enrichment adds those dimensions at ingest: a date to sort by, a section path to disambiguate, an entity to join on. The risk is that enrichment is inference, and an inferred date or author that is wrong will filter out the correct document with no error to show for it.

Mental model

Enrichment is a catalogue card written for each document: author, date, subject, shelf. The card makes the library navigable, and it also becomes the thing people trust instead of the book, even when the card was filled in hastily by someone who only skimmed the cover.

How it works

Mechanism, step by stepInteractive
  1. 1List the fields the product actually queries on, such as date, type, owner, section and entity, and drop the rest.
  2. 2Prefer deterministic extraction from document structure, such as headings and document properties, before model inference.
  3. 3Store each field with its type, its source span and a confidence, so a wrong value can be traced.
  4. 4Normalise dates and identifiers to one format at ingest rather than in every query.
  5. 5Backfill new fields with a versioned enrichment run instead of mixing old and new semantics in one index.
  6. 6Measure field accuracy on a sample and gate any field used in a filter behind a minimum precision.

Step 1 of 6

List the fields the product actually queries on, such as date, type, owner, section and entity, and drop the rest.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. List the fields the product actually queries on, such as date, type, owner, section and e… 2. Prefer deterministic extraction from document structure, such as headings and document pr… 3. Store each field with its type, its source span and a confidence, so a wrong value can be… 4. Normalise dates and identifiers to one format at ingest rather than in every query. 5. Backfill new fields with a versioned enrichment run instead of mixing old and new semanti… 6. Measure field accuracy on a sample and gate any field used in a filter behind a minimum p…

  1. List the fields the product actually queries on, such as date, type, owner, section and entity, and drop the rest.
  2. Prefer deterministic extraction from document structure, such as headings and document properties, before model inference.
  3. Store each field with its type, its source span and a confidence, so a wrong value can be traced.
  4. Normalise dates and identifiers to one format at ingest rather than in every query.
  5. Backfill new fields with a versioned enrichment run instead of mixing old and new semantics in one index.
  6. Measure field accuracy on a sample and gate any field used in a filter behind a minimum precision.

Trade-offs

ChoiceGood forCostFails when
Structural extraction onlyDeterministic and cheapMisses fields the layout does not exposeDocuments vary widely in format
LLM-generated metadataHandles varied formats and free textCost, latency and hallucinated valuesThe field drives a hard filter
Rich enrichmentBetter filtering and rankingIngest cost and schema churnThe corpus is large and changes constantly
Sparse enrichmentSimple schemaQueries fall back to text matchingUsers need date or ownership constraints
Trade-off mapInteractive

Good for

Deterministic and cheap

Cost

Misses fields the layout does not expose

Fails when

Documents vary widely in format

  1. Documents vary widely in format
  2. The field drives a hard filter
  3. The corpus is large and changes constantly
  4. Users need date or ownership constraints

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Structural extraction only is good for Deterministic and cheap, costs Misses fields the layout does not expose, and fails when Documents vary widely in format. LLM-generated metadata is good for Handles varied formats and free text, costs Cost, latency and hallucinated values, and fails when The field drives a hard filter. Rich enrichment is good for Better filtering and ranking, costs Ingest cost and schema churn, and fails when The corpus is large and changes constantly. Sparse enrichment is good for Simple schema, costs Queries fall back to text matching, and fails when Users need date or ownership constraints

Failure modes

  • Hallucinated document date. A model infers a plausible date from prose and the document is filtered out of every recent query. Debug: compare enriched dates with the source metadata for a sample and count mismatches.
  • Inconsistent field semantics. Old documents carry a section name that was renamed, so filters split one section into two. Debug: list distinct values per field and look for near-duplicates.
  • Enrichment blocks ingestion. A slow model call per document makes the pipeline lag and the index go stale. Debug: measure enrichment time per document and compare it with the ingest rate.

In production

Track per-field null rate, distinct value count and drift over time. Cap enrichment latency per document and fall back to structural fields when the model is slow.

Common misconceptions

People think: More metadata always improves retrieval.

Actually: Unused fields cost ingest time and create filter traps; only fields that queries use earn their place.

People think: Extracted metadata is ground truth.

Actually: It is a prediction with a source and a confidence, and it should be treated like any other model output.

Interview angle

Begin from the queries the product must support, then derive the minimal schema. Distinguish structural extraction from model inference, and mention confidence and backfill. Proposing to extract everything with a model, without discussing wrong values filtering out correct documents, is the over-engineered answer.

Sources