Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 23 of 57

Chunking

0 of 57 complete

Why it matters

Retrieval does not return documents, it returns chunks. That makes chunking the point where document structure is either preserved or destroyed, and every later stage inherits the decision. A chunk that begins mid-sentence, or a table separated from the heading that names its column, cannot be fixed by a better embedding model or a reranker — the missing context was never in the candidate.

It is also the cheapest place to improve a RAG system. Changing chunk size and adding section headers usually moves retrieval quality more than swapping models, and it requires no new infrastructure. The cost is that re-chunking means re-embedding the corpus, so the decision has migration weight and should be based on measured retrieval recall rather than intuition.

Mental model

Chunking is cutting a long document into index cards, one fact per card, each card still readable on its own. A reader handed one card should know what it is about without the preceding pages.

The metaphor breaks on two points. Real documents have dependencies — a pronoun refers to a name three paragraphs up, a table's units live in a caption — so a card that reads cleanly to you may be ambiguous to a model that only sees the card. And the right card size depends on the question: "what is the refund window" wants a sentence, "summarise the refund policy" wants the section. That tension is why production systems often keep two granularities at once.

How it works

  1. Parse before splitting. Extract text with structure intact: headings, list markers, table cells, page numbers, code blocks. Splitting a parsed tree is far better than splitting a flat string, because structure tells you where a natural boundary is.
  2. Choose a primary boundary. Prefer document hierarchy — section, subsection, then paragraph. Fall back to sentence boundaries, then to a token window. Never split inside a table row, a code block, or a numbered list item.
  3. Set a target size. Aim for a few hundred tokens, small enough that a vector represents one idea, large enough to answer a question. Enforce a hard maximum below the embedding model's input limit; exceeding it truncates silently.
  4. Add overlap, deliberately. A small overlap (in the region of 10–15% of the chunk) protects answers that straddle a boundary. Overlap is not free: it multiplies storage and produces near-duplicate hits.
  5. Attach context. Prepend the document title and section path. Optionally generate a one-line summary of what the chunk covers and prepend that too — the "contextual retrieval" pattern.
  6. Store provenance. Keep document ID, version, character offsets, effective date and ACLs with each chunk so citations can highlight the source and filters can enforce permissions.
  7. Evaluate, then adjust. Build a labelled query set, compute recall@k for each chunking configuration, and pick on evidence.

The boundary arithmetic is worth doing once. With chunk size ss and stride d=s−od = s - o (overlap oo), a span of length a≤sa \le s is fully contained in some chunk with probability approximately

P(contained)≈min⁡(1,s−ad)P(\text{contained}) \approx \min\left(1, \frac{s - a}{d}\right)

So a 400-token chunk with 100-token overlap has a stride of 300; a 100-token answer is contained roughly 100% of the time, while a 250-token answer only about half the time. Larger answers need larger parent units, not more overlap.

Chunk walls on the pageInteractive

3 chunks · 377 chunk characters (includes repeats) · table split

chunk regionoverlap: repeated tailwall at chunk starttable split

Every wall is at a fixed character count. The boundaries ignore words, sentences, headings and the table's structure.

Hover a tinted region or a wall on the page, or pick a chunk below, to read the text it contains. Flags mark where each chunk starts.

! The header row is cut between chunks 2 and 3.

! Its data rows are in chunk 3.

Figure. Five strategies over the same refund policy page. Size and overlap redraw the walls where those controls apply; overlap repeats the previous tail, so the same characters belong to two chunks. Fixed size is the default counterexample; Recursive and Sentence keep document structure where they can. The figure shows boundaries and the extra characters overlap adds, not retrieval quality: matching precision, readability and duplicate hits are not measured here.

What to notice. Fixed windows cut on character count, so words and sentences are severed and a table row can land in a different chunk from its header, or be cut mid-row. The in-place warning marks both cases. Characters are the unit here; a real pipeline would count tokens with the embedding model's tokenizer.

Text version

A refund policy page shown as extracted monospace text: a heading, two paragraphs, a three-row table with Plan, Window and Method columns, and a three-line code block. Tinted bands cover the characters belonging to each chunk, and a numbered flag with a vertical wall marks each chunk start in the left margin. Overlap is a hatched band over the last characters of the previous chunk, which are repeated at the start of the next. The table shows a warning marker when a row is cut mid-row or its header is separated from its data rows. Parent–child replaces the walls with one large bracket per page region and smaller brackets for the children inside it.

A minimal implementation

import re
 
def split_by_heading(text: str) -> list[dict]:
    """Split on headings, then blank-line paragraphs. Fenced code must be passed through whole by the caller."""
    sections = re.split(r"(?m)^(#{1,4} .*)$", text)
    chunks, heading = [], ""
    for part in sections:
        if re.match(r"^#{1,4} ", part):
            heading = part.strip("# ").strip()
            continue
        for para in [p for p in part.split("\n\n") if p.strip()]:
            prefix = f"{heading}\n" if heading else ""
            chunks.append({"text": prefix + para.strip(), "heading": heading})
    return chunks
 
def check_sizes(chunks: list[dict], max_words: int = 400) -> None:
    # Words are a cheap proxy; run the real tokenizer before embedding.
    for c in chunks:
        if len(c["text"].split()) > max_words:
            raise ValueError(f"chunk exceeds {max_words} words: {c['heading']!r}")
 
def contextualise(chunk: dict, doc_title: str) -> str:
    # The title and section path are the cheapest disambiguation available.
    return f"{doc_title} > {chunk['heading']}\n\n{chunk['text']}"

Real pipelines also handle tables (keep header plus rows together, or emit one chunk per row with the header repeated) and PDFs (require layout-aware parsing before any of this is meaningful).

Trade-offs

StrategyGood forCostFails when
Fixed-size windowsUniform corpora, quick baselinesBreaks sentences and tablesDocuments have strong structure
Structure-aware splitMarkdown, HTML, contracts, manualsDepends on parser qualitySource is a scanned PDF with no reliable layout
Semantic breakpointsLong prose with drifting topicsEmbedding pass at ingest, unstable boundariesCorpus is short or already well sectioned
Parent-childPrecise matching with readable contextTwo indexes or a lookup storeStorage and ingestion complexity dominate
Contextual prefixesAmbiguous, pronoun-heavy documentsExtra model calls or summaries per chunkCorpus is huge and ingestion budget is fixed
Trade-off mapInteractive

Good for

Uniform corpora, quick baselines

Cost

Breaks sentences and tables

Fails when

Documents have strong structure

  1. Documents have strong structure
  2. Source is a scanned PDF with no reliable layout
  3. Corpus is short or already well sectioned
  4. Storage and ingestion complexity dominate
  5. Corpus is huge and ingestion budget is fixed

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Fixed-size windows is good for Uniform corpora, quick baselines, costs Breaks sentences and tables, and fails when Documents have strong structure. Structure-aware split is good for Markdown, HTML, contracts, manuals, costs Depends on parser quality, and fails when Source is a scanned PDF with no reliable layout. Semantic breakpoints is good for Long prose with drifting topics, costs Embedding pass at ingest, unstable boundaries, and fails when Corpus is short or already well sectioned. Parent-child is good for Precise matching with readable context, costs Two indexes or a lookup store, and fails when Storage and ingestion complexity dominate. Contextual prefixes is good for Ambiguous, pronoun-heavy documents, costs Extra model calls or summaries per chunk, and fails when Corpus is huge and ingestion budget is fixed

Failure modes

  • Orphaned heading. A table of prices is indexed without the plan name in its heading, so every row looks identical. Debug: open the raw chunk text for a failing query and read it cold; if you cannot tell what it describes, neither can the retriever.
  • Split answer. The question needs two adjacent sentences and they landed in different chunks, so neither is convincing alone. Debug: check whether the correct answer spans a boundary by locating its offsets; if it does, reduce stride or switch to parent-child.
  • Near-duplicate flooding. Overlap plus repeated boilerplate fills the top results with five versions of one sentence. Debug: group the top 20 results by near-identical text and count distinct documents; apply deduplication or MMR.
  • Silent truncation. Chunks exceed the embedding model's limit and the tail is dropped at encode time. Debug: assert chunk token length against the model maximum during ingestion and fail the job rather than truncating.

In production

Chunking is an ingestion-time cost and a re-indexing risk. Version the chunking configuration alongside the embedding model, so a chunk in the index can be traced to the code that produced it. Store character offsets so citations can highlight the exact sentence, and keep document versions so old chunks can be deleted rather than shadowing new ones.

Monitor the distribution of chunk sizes, the fraction of chunks that fail a "does this stand alone" check, duplicate rates per document, and recall@k on the labelled set after every re-chunk. A jump in median chunk size usually means the parser changed.

Common misconceptions

People think: Smaller chunks always improve retrieval.

Actually: Smaller chunks sharpen matching but strip context, so the passage may match the query and still not answer it. Precision without sufficiency just returns confident fragments.

People think: Overlap is free insurance.

Actually: Overlap multiplies storage and embeddings, and generates near-duplicate results that crowd the context. It only helps spans that straddle boundaries.

Interview angle

The standard question is "how do you choose a chunk size", and the weak answer is a number. The strong answer describes the retrieval unit versus the generation unit, names structure-aware splitting, explains what breaks at the extremes, and says how they would measure it — recall@k on labelled queries, plus an inspection of whether retrieved chunks are self-contained. See chunking interview questions and chunking strategies.

Sources