Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 27 of 57

Contextual retrieval

0 of 57 complete

Why it matters

Chunking removes exactly the information that makes a sentence interpretable: the heading, the entity, the section. A dense vector built from the fragment encodes the ambiguity, so a query naming the real subject does not match. Contextual retrieval restores the missing frame by writing a few sentences that situate the chunk in its document and embedding that alongside the text.

Mental model

It is annotating each page of a photocopied report with a margin note saying which chapter it came from. The note makes the page findable. The metaphor breaks because the note is generated, so it can assert a subject the page does not actually discuss.

How it works

Mechanism, step by stepInteractive
  1. 1Parse the document and keep its title and outline available to the context generator.
  2. 2For each chunk, generate a short situating passage that names the document, the section and the entity the chunk refers to.
  3. 3Prepend that passage to the chunk text and embed the concatenation, caching the generated context per chunk.
  4. 4Store the original text separately so citations quote the source and not the generated frame.
  5. 5Apply the same contextualisation to the lexical index if you run hybrid retrieval, since identifiers benefit too.
  6. 6Evaluate recall with and without context on queries that use the document's subject rather than the chunk's wording.

Step 1 of 6

Parse the document and keep its title and outline available to the context generator.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Parse the document and keep its title and outline available to the context generator. 2. For each chunk, generate a short situating passage that names the document, the section a… 3. Prepend that passage to the chunk text and embed the concatenation, caching the generated… 4. Store the original text separately so citations quote the source and not the generated fr… 5. Apply the same contextualisation to the lexical index if you run hybrid retrieval, since … 6. Evaluate recall with and without context on queries that use the document's subject rathe…

  1. Parse the document and keep its title and outline available to the context generator.
  2. For each chunk, generate a short situating passage that names the document, the section and the entity the chunk refers to.
  3. Prepend that passage to the chunk text and embed the concatenation, caching the generated context per chunk.
  4. Store the original text separately so citations quote the source and not the generated frame.
  5. Apply the same contextualisation to the lexical index if you run hybrid retrieval, since identifiers benefit too.
  6. Evaluate recall with and without context on queries that use the document's subject rather than the chunk's wording.

Trade-offs

ChoiceGood forCostFails when
Generated context per chunkRecovers subject and pronoun referencesOne model call per chunk at ingestThe corpus is large and ingest budget is fixed
Structural context onlyDeterministic and freeCannot resolve references the structure does not encodeDocuments use implicit subjects throughout
Context on children onlyCheap and targetedParent lookups still lack framingThe pipeline returns raw parents
No contextualisationSimplest ingestFragments embed as ambiguous statementsChunks are short and pronoun-heavy
Trade-off mapInteractive

Good for

Recovers subject and pronoun references

Cost

One model call per chunk at ingest

Fails when

The corpus is large and ingest budget is fixed

  1. The corpus is large and ingest budget is fixed
  2. Documents use implicit subjects throughout
  3. The pipeline returns raw parents
  4. Chunks are short and pronoun-heavy

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Generated context per chunk is good for Recovers subject and pronoun references, costs One model call per chunk at ingest, and fails when The corpus is large and ingest budget is fixed. Structural context only is good for Deterministic and free, costs Cannot resolve references the structure does not encode, and fails when Documents use implicit subjects throughout. Context on children only is good for Cheap and targeted, costs Parent lookups still lack framing, and fails when The pipeline returns raw parents. No contextualisation is good for Simplest ingest, costs Fragments embed as ambiguous statements, and fails when Chunks are short and pronoun-heavy

Failure modes

  • Context contradicts the chunk. The generator invents a subject and the chunk becomes retrievable for the wrong topic. Debug: sample generated contexts and check each names an entity that appears in the parent document.
  • Context dominates the vector. The prepended frame is longer than the chunk, so all chunks of a document become near-duplicates. Debug: compare pairwise similarity within one document before and after contextualisation.
  • Context not cached. A retry regenerates different context and produces vectors that no longer match the index. Debug: hash the generated context per chunk and confirm it is stable across runs.

In production

Cache generated context keyed by chunk id and document version, and rebuild it when the document changes. Track ingest cost per document and the similarity spread within a document, which collapses if context overwhelms the text.

Common misconceptions

People think: Contextual retrieval is just adding the title to every chunk.

Actually: The value comes from resolving the specific references in that chunk, which a title alone does not do.

People think: It replaces the need for good chunk boundaries.

Actually: It compensates for fragments; chunks that mix topics still produce diluted vectors.

Interview angle

Explain the ambiguity chunking introduces and why the fix belongs at index time. Keep generated context separate from stored source text for citation, and mention caching. Prepending the document title and stopping there misses the specific reference resolution that gives the technique its value.

Sources