Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 25 of 57

Overlap

0 of 57 complete

Why it matters

Any hard cut can separate a question's setup from its answer. Overlap hedges against that by letting neighbouring chunks share a tail and a head, so the probability that some chunk contains the whole answer rises. The cost is structural: the index stores the same text several times, near-duplicates compete for the same slots, and citations point at repeated spans.

Mental model

Overlap is laying roof tiles so each row covers part of the one below. Water does not reach the seam. The metaphor breaks because overlapping tiles do not change the roof's weight much, while overlapping chunks multiply storage and duplicate evidence in the prompt.

How it works

Overlap bought and paid forInteractive

Retrieved context · illustrative tokens

1,960 input+ 500 reserve= 2,460 of 2,800

340 free

Step 1/4 · No overlap

Step 1No overlap

Four unique chunks and 340 tokens free. The refund window starts in one chunk and the exception list opens in the next, so no retrieved unit contains the whole answer.

reserve 5001,960 tokens

Output reserve

500

The answer needs both spans: 'Refund window' holds the rule, 'Exception list' the list it refers to.

output reservefixedBar length is each block's token share of the window.

Pick a section to read its token count, its role and its state.

Figure. Overlap is insurance against a cut boundary, charged to the same window. With none, the answer is split across two chunks; 500 tokens of repeated tails make it whole; heavy overlap spends 750 tokens on repeats and crowds the evidence slots.

What to notice. Counts are authored for a 2,800-token window with a 500-token answer reserve; no chunker or retriever was run. The figure shows the retrieved set in the context, not index size or retrieval scores. Deduplication is drawn as removal; in a real index the duplicate vectors remain and are filtered at query time. Chunk boundaries inside the source document are not shown.

Text version

A 2,800-token window with a 500-token answer reserve, so 2,300 tokens are available. The default first step has four 420-token chunks, instructions 200 and question 80: 1,960 input tokens and 340 free. The refund window starts in one chunk and its exception list in the next. The second step adds two 250-token repeated tails, one after each of those chunks, for 2,460 input tokens — 160 past the cap rule and into the reserve. The third step applies heavy overlap with three repeats, for 2,710 tokens, 410 over the window. The fourth step removes all three repeats and returns to four unique spans at 1,960 tokens and 340 free. Selecting a chunk or a repeat shows its count and its role.

  1. Split on structure first, and add overlap only where a hard boundary falls inside a sentence or a table.
  2. Set overlap from sentence length rather than a percentage, so the repeated tail contains complete sentences.
  3. Keep the overlap region out of the chunk's citation span, or record both the chunk span and the unique span.
  4. Measure index growth against the duplication factor and confirm the storage budget absorbs it.
  5. At retrieval, collapse chunks that share a parent and overlapping spans before filling the context.
  6. Test with questions whose answers sit near a known boundary, and compare against zero overlap.

Trade-offs

ChoiceGood forCostFails when
No overlapSmallest index and no duplicatesBoundary answers can be lostChunks are cut by fixed length rather than structure
Sentence-level overlapProtects complete sentences cheaplyModest index growthAnswers span several paragraphs
Heavy overlapHigh chance the answer is intact somewhereIndex size and duplicate results grow sharplyTop-k is small and duplicates waste slots
Parent-child instead of overlapContext without duplicationNeeds a parent store and a two-level lookupThe pipeline cannot store parent documents
Trade-off mapInteractive

Good for

Smallest index and no duplicates

Cost

Boundary answers can be lost

Fails when

Chunks are cut by fixed length rather than structure

  1. Chunks are cut by fixed length rather than structure
  2. Answers span several paragraphs
  3. Top-k is small and duplicates waste slots
  4. The pipeline cannot store parent documents

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: No overlap is good for Smallest index and no duplicates, costs Boundary answers can be lost, and fails when Chunks are cut by fixed length rather than structure. Sentence-level overlap is good for Protects complete sentences cheaply, costs Modest index growth, and fails when Answers span several paragraphs. Heavy overlap is good for High chance the answer is intact somewhere, costs Index size and duplicate results grow sharply, and fails when Top-k is small and duplicates waste slots. Parent-child instead of overlap is good for Context without duplication, costs Needs a parent store and a two-level lookup, and fails when The pipeline cannot store parent documents

Failure modes

  • Duplicate results in the prompt. Three overlapping chunks of the same passage occupy the context and crowd out other evidence. Debug: group the retrieved chunks by parent id and overlap of spans, then count distinct regions.
  • Index bloat from a global percentage. A large overlap applied to short chunks nearly doubles storage with little benefit. Debug: compare chunk count and total tokens with overlap disabled.
  • Overlap masking a parse error. Repeated headers or footers land inside the overlap and appear in several chunks. Debug: inspect the shared prefix of adjacent chunks for boilerplate.

In production

Track chunks per document and the duplicate rate in retrieved sets. If duplicates exceed a small fraction of the top-k, reduce overlap or add span-level deduplication.

Common misconceptions

People think: Overlap improves retrieval quality directly.

Actually: It only reduces the chance that a boundary destroys the answer; it adds no new information to the index.

People think: A fixed overlap percentage works for every corpus.

Actually: Overlap should be measured in sentences and tied to chunk size, not applied as a constant fraction.

Interview angle

Present overlap as insurance against bad boundaries, priced in index size and duplicate context. Prefer structural splitting and mention span-level deduplication. Stating a percentage as best practice, with no account of what it protects against, is the weak form of this answer.

Sources