juniorReportedcurated
How do you choose a chunk size, and what concretely goes wrong at each extreme?
30-second answer
- Small chunks of 128-256 tokens give sharp embeddings and precise matches but sever context, leaving a fragment that says the limit is 30 days with no idea what limit.
- Large chunks keep context, but one vector averages several topics, so it matches nothing specifically and burns prompt tokens.
- Prose usually lands at 300-800 tokens with 10-15% overlap, while tables and code stay whole.
- The honest answer is a sweep: index at 200, 400, 800 and 1,200 tokens and compare recall@k and end-to-end correctness on labelled queries.
- Smaller chunks raise recall but increase chunk count, embedding cost and duplicate hits, and re-chunking invalidates every stored vector.
Likely follow-ups
Open a question to rehearse the answer.
Which metric tells you a chunk is too large, and what does the failure look like in the retrieved text?
- Recall@k falls while retrieved chunks stay on-topic, because the vector averaged several topics and matches none of them precisely.
- Read the chunk cold: if you cannot tell what it describes, or it covers three subjects, it is too large for that question distribution.
- Check whether the answer span is a small fraction of the chunk, which dilutes the match and spends prompt tokens on unused text.
How does adding a reranker change your optimal chunk size, does the sweep result move?
- The stage buys precision at the top, so it lets children stay small while the reranker filters the shortlist, which moves the sweep optimum rather than simply shrinking chunks.
- A cross-encoder scores the passage it is given, so a child stripped of its heading is scored on a fragment and the correct hit can be demoted.
- Keep the heading attached to each child and pair small children with parent expansion, so the matching unit is not the only unit the model reads.
- Re-run the sweep after adding the stage, since the optimum moves with the pipeline rather than staying a constant.
How do you handle documents whose natural units sit far outside 128-800 tokens, like a 40-page contract?
- Split on structure first, clause then section then sub-clause, accepting units outside the range where the document demands it.
- Use parent-child so a clause is matched and the surrounding section is read, which is the case a fixed-size sweep cannot cover.
- Never split inside a table row, code block or numbered list item, and enforce a hard maximum below the embedding model's input limit.
Red flags
- universal chunk size
- ignores model max length
Concepts this tests
Tags
Source: saadho.arushk.tech
More in Data & storage
- Users say a rate-limits answer quotes correct numbers but never says what they apply to. You trace it to one chunk. Diagnose and fix.
- How do you ingest a 200-page PDF of charts and tables where the answer exists only inside a line chart, with no matching text?
- Page 1 of a financial report says "all amounts in thousands". How do you stop chunks from losing that document-wide context?