Why it matters
Raw chunk text cannot answer questions about recency, ownership or document type, and it cannot be filtered without scanning. Enrichment adds those dimensions at ingest: a date to sort by, a section path to disambiguate, an entity to join on. The risk is that enrichment is inference, and an inferred date or author that is wrong will filter out the correct document with no error to show for it.
Mental model
Enrichment is a catalogue card written for each document: author, date, subject, shelf. The card makes the library navigable, and it also becomes the thing people trust instead of the book, even when the card was filled in hastily by someone who only skimmed the cover.
How it works
- 1List the fields the product actually queries on, such as date, type, owner, section and entity, and drop the rest.
- 2Prefer deterministic extraction from document structure, such as headings and document properties, before model inference.
- 3Store each field with its type, its source span and a confidence, so a wrong value can be traced.
- 4Normalise dates and identifiers to one format at ingest rather than in every query.
- 5Backfill new fields with a versioned enrichment run instead of mixing old and new semantics in one index.
- 6Measure field accuracy on a sample and gate any field used in a filter behind a minimum precision.
Step 1 of 6
List the fields the product actually queries on, such as date, type, owner, section and entity, and drop the rest.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. List the fields the product actually queries on, such as date, type, owner, section and e… 2. Prefer deterministic extraction from document structure, such as headings and document pr… 3. Store each field with its type, its source span and a confidence, so a wrong value can be… 4. Normalise dates and identifiers to one format at ingest rather than in every query. 5. Backfill new fields with a versioned enrichment run instead of mixing old and new semanti… 6. Measure field accuracy on a sample and gate any field used in a filter behind a minimum p…
- List the fields the product actually queries on, such as date, type, owner, section and entity, and drop the rest.
- Prefer deterministic extraction from document structure, such as headings and document properties, before model inference.
- Store each field with its type, its source span and a confidence, so a wrong value can be traced.
- Normalise dates and identifiers to one format at ingest rather than in every query.
- Backfill new fields with a versioned enrichment run instead of mixing old and new semantics in one index.
- Measure field accuracy on a sample and gate any field used in a filter behind a minimum precision.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Structural extraction only | Deterministic and cheap | Misses fields the layout does not expose | Documents vary widely in format |
| LLM-generated metadata | Handles varied formats and free text | Cost, latency and hallucinated values | The field drives a hard filter |
| Rich enrichment | Better filtering and ranking | Ingest cost and schema churn | The corpus is large and changes constantly |
| Sparse enrichment | Simple schema | Queries fall back to text matching | Users need date or ownership constraints |
Good for
Deterministic and cheap
Cost
Misses fields the layout does not expose
Fails when
Documents vary widely in format
- Documents vary widely in format
- The field drives a hard filter
- The corpus is large and changes constantly
- Users need date or ownership constraints
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Structural extraction only is good for Deterministic and cheap, costs Misses fields the layout does not expose, and fails when Documents vary widely in format. LLM-generated metadata is good for Handles varied formats and free text, costs Cost, latency and hallucinated values, and fails when The field drives a hard filter. Rich enrichment is good for Better filtering and ranking, costs Ingest cost and schema churn, and fails when The corpus is large and changes constantly. Sparse enrichment is good for Simple schema, costs Queries fall back to text matching, and fails when Users need date or ownership constraints
Failure modes
- Hallucinated document date. A model infers a plausible date from prose and the document is filtered out of every recent query. Debug: compare enriched dates with the source metadata for a sample and count mismatches.
- Inconsistent field semantics. Old documents carry a section name that was renamed, so filters split one section into two. Debug: list distinct values per field and look for near-duplicates.
- Enrichment blocks ingestion. A slow model call per document makes the pipeline lag and the index go stale. Debug: measure enrichment time per document and compare it with the ingest rate.
In production
Track per-field null rate, distinct value count and drift over time. Cap enrichment latency per document and fall back to structural fields when the model is slow.
Common misconceptions
People think: More metadata always improves retrieval.
Actually: Unused fields cost ingest time and create filter traps; only fields that queries use earn their place.
People think: Extracted metadata is ground truth.
Actually: It is a prediction with a source and a confidence, and it should be treated like any other model output.
Interview angle
Begin from the queries the product must support, then derive the minimal schema. Distinguish structural extraction from model inference, and mention confidence and backfill. Proposing to extract everything with a model, without discussing wrong values filtering out correct documents, is the over-engineered answer.
Sources
- PostgreSQL, JSON functions and operators — storing and querying typed metadata alongside vector columns.
- JSON Schema specification — validating the shape of enriched metadata before it reaches the index.
- Elasticsearch, Mapping — declaring field types so metadata is filterable rather than free text.