Why it matters
Embeddings compress a span of text into one point, so a chunk that covers three topics is a point between them and matches none of them well. Cut too small and the answer loses the sentence that explains it. Chunking strategy is the attempt to align the retrieval unit with the unit of meaning, using whatever structure the document offers before falling back to length.
Mental model
Chunking is cutting a long film into scenes. Cut at scene boundaries and each clip stands alone; cut every thirty seconds and you split conversations in half. The metaphor breaks because a document rarely marks its scenes as clearly as a film does, so some boundaries must be guessed.
How it works
Retrieved context · illustrative tokens
2,200 input+ 600 reserve= 2,800 of 3,600
800 free
Step 1Fixed windows
Four 500-token windows plus instructions and question come to 2,200 tokens. The refund rule starts in window 2 and its exception list opens in window 3, so neither chunk holds the whole answer.
Output reserve
600The refund rule starts in window 2 and the exception list in window 3; neither chunk holds both.
Pick a section to read its token count, its role and its state.
Figure. The retrieval unit is the context unit: four 500-token windows fit neatly and split the answer, structure-aware chunks are uneven and keep it whole, and parent–child spends 1,400 extra tokens to restore the paragraph around a precise match.
What to notice. Counts are authored for a 3,600-token window with a 600-token answer reserve; no document was parsed and no embedding model was run. The figure shows how retrieved chunks occupy the window, not retrieval scores or index size. Parse quality, vector count and the questions used to choose a size are omitted.
Text version
A 3,600-token window with a 600-token answer reserve, so 3,000 tokens are available. The default first step is four fixed 500-token windows plus instructions 120 and question 80: 2,200 input tokens and 800 free. The refund rule starts in the second window and its exception list lands in the third, so neither chunk holds the whole answer. The second step replaces them with heading-aligned chunks of 900, 640 and 520 tokens; the 640-token chunk holds the whole answer. The third step is a semantic split of 820, 460 and 940 tokens. The fourth step is parent–child: three children of 150 tokens used for matching, with a switch labelled Generation unit. With children only the input is 650 tokens; adding the 1,400-token parent brings it to 2,050. Selecting a chunk shows its size and role.
- Parse first and keep the heading tree, because structure-aware splitting depends on knowing where sections begin.
- Split on the largest structural boundary that fits the token budget: document, section, subsection, paragraph.
- Fall back to sentence boundaries inside a section rather than cutting mid-sentence at a character count.
- Choose a target size from the question distribution: answers that span a paragraph need chunks that contain a paragraph.
- Attach the section path and document title to each chunk so a fragment keeps its context after splitting.
- Compare strategies on a labelled set by measuring whether the gold answer is fully contained in the retrieved chunk.
4 chunks · ribbon width is proportional to characters, not document position
Use when. Documents have paragraphs and headings; you want size limits without mid-sentence cuts.
Fails when. One paragraph is longer than the limit and must be split anyway.
Model note. Semantic is a hand-built two-topic grouping here; a production system uses embedding similarity and a break threshold. Parent–child children are half the size setting.
Token counts are an approximation (characters ÷ 4). “Semantic” here groups the four paragraphs into two topics by hand; a production system uses embedding similarity and a break threshold. Parent–child shows each parent’s text above its children; children are half the size slider, which is why the strategy always splits.
Figure. Five strategies over the same policy document. Change the size and overlap and watch where the boundaries land.
What to notice. Fixed size cuts mid-sentence. Recursive respects paragraphs until one is too long. Semantic here is a hand-built two-topic grouping, not an embedding model. Parent–child keeps a large parent for generation and small children for matching.
Text version
A refund policy is split by five strategies. Fixed size produces equal character blocks, some cutting sentences. Recursive keeps paragraphs and only splits when a paragraph exceeds the limit. Sentence groups whole sentences. Semantic groups whole paragraphs by topic. Parent-child shows four parents, each split into smaller children. Sliders change size and overlap; overlap repeats the tail of the previous chunk.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Fixed token windows | Uniform size and simple batching | Cuts across headings and sentences | Documents have strong section structure |
| Structure-aware splitting | Chunks align with meaning | Uneven sizes and dependence on parse quality | Parsing loses headings or the source is plain text |
| Semantic splitting | Boundaries follow topic shifts | Extra embedding pass and variable lengths | The corpus is large and ingest time is constrained |
| Proposition-level chunks | Very precise matching | Many more chunks and lost surrounding context | Questions need multi-sentence reasoning |
Good for
Uniform size and simple batching
Cost
Cuts across headings and sentences
Fails when
Documents have strong section structure
- Documents have strong section structure
- Parsing loses headings or the source is plain text
- The corpus is large and ingest time is constrained
- Questions need multi-sentence reasoning
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Fixed token windows is good for Uniform size and simple batching, costs Cuts across headings and sentences, and fails when Documents have strong section structure. Structure-aware splitting is good for Chunks align with meaning, costs Uneven sizes and dependence on parse quality, and fails when Parsing loses headings or the source is plain text. Semantic splitting is good for Boundaries follow topic shifts, costs Extra embedding pass and variable lengths, and fails when The corpus is large and ingest time is constrained. Proposition-level chunks is good for Very precise matching, costs Many more chunks and lost surrounding context, and fails when Questions need multi-sentence reasoning
Failure modes
- Answer split across a boundary. The retrieved chunk contains the setup but the number sits in the next chunk. Debug: for failing questions, check whether the gold span crosses a chunk boundary.
- Heading orphaned from its section. A chunk begins with body text whose heading lives in another chunk, so the fragment has no subject. Debug: print the first line of random chunks and look for sentences with unresolved pronouns or missing subjects.
- Chunk size set by token budget alone. Chunks fit the model but contain several unrelated topics, so their vectors sit between subjects. Debug: inspect the nearest neighbours of a chunk; unrelated results suggest a diluted vector.
In production
Version the chunker with the index, because changing boundaries invalidates stored vectors and citations. Track the distribution of chunk lengths and the share of chunks with no heading attached.
Common misconceptions
People think: There is a correct chunk size.
Actually: The right unit follows the questions and the document structure; it is an empirical choice, not a constant.
People think: Smaller chunks always retrieve more precisely.
Actually: They match more precisely and carry less meaning, often retrieving a fragment that cannot answer the question.
Interview angle
Frame chunking as aligning the retrieval unit with the unit of meaning, then list the structural signals and the fallback order. Describe measuring answer containment, not just recall. Quoting a chunk size and an overlap percentage with no reference to the corpus or the questions is the answer that signals memorised defaults.
Sources
- Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey — chunking granularity as a retrieval design choice.
- Chen et al., Dense X Retrieval — proposition-level units and the effect of granularity on retrieval.
- Lewis et al., Retrieval-Augmented Generation — the passage-as-unit assumption behind retrieval-augmented generation.