Skip to content
AI Engineer Guide.
juniorReportedcurated

How do you choose a chunk size, and what concretely goes wrong at each extreme?

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

Which metric tells you a chunk is too large, and what does the failure look like in the retrieved text?
  • Recall@k falls while retrieved chunks stay on-topic, because the vector averaged several topics and matches none of them precisely.
  • Read the chunk cold: if you cannot tell what it describes, or it covers three subjects, it is too large for that question distribution.
  • Check whether the answer span is a small fraction of the chunk, which dilutes the match and spends prompt tokens on unused text.
How does adding a reranker change your optimal chunk size, does the sweep result move?
  • The stage buys precision at the top, so it lets children stay small while the reranker filters the shortlist, which moves the sweep optimum rather than simply shrinking chunks.
  • A cross-encoder scores the passage it is given, so a child stripped of its heading is scored on a fragment and the correct hit can be demoted.
  • Keep the heading attached to each child and pair small children with parent expansion, so the matching unit is not the only unit the model reads.
  • Re-run the sweep after adding the stage, since the optimum moves with the pipeline rather than staying a constant.
How do you handle documents whose natural units sit far outside 128-800 tokens, like a 40-page contract?
  • Split on structure first, clause then section then sub-clause, accepting units outside the range where the document demands it.
  • Use parent-child so a clause is matched and the surrounding section is read, which is the case a fixed-size sweep cannot cover.
  • Never split inside a table row, code block or numbered list item, and enforce a hard maximum below the embedding model's input limit.

Red flags

  • universal chunk size
  • ignores model max length

Concepts this tests

Tags

Source: saadho.arushk.tech

More in Data & storage