Why it matters
A code question is usually about a symbol — where it is defined, who calls it, what changed. Line-based chunking cuts functions in half and produces fragments that neither embed well nor read well. Code retrieval parses the repository into a syntax tree, chunks at declaration boundaries, and keeps the file path and symbol name as metadata so a hit can be resolved to a location a developer can open.
Mental model
Code retrieval is a symbol index in a compiler rather than a concordance: it knows that this name is declared here and used there. The metaphor breaks because the index is built for search, not type checking, so it can find a name without knowing which of several definitions is the one that runs.
How it works
- 1Parse each file with a grammar for its language and extract declarations with their spans and parent scopes.
- 2Chunk at declaration boundaries, keeping the signature, docstring and body together.
- 3Store path, language, symbol name and enclosing scope as metadata for every chunk.
- 4Index identifiers in a lexical field as well as a dense field, since exact symbol names are the common query.
- 5Include call sites and tests as separate chunk types so a query can ask for usage rather than definition.
- 6Re-index on commit or on a schedule, and drop chunks for deleted files to avoid stale definitions.
Step 1 of 6
Parse each file with a grammar for its language and extract declarations with their spans and parent scopes.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Parse each file with a grammar for its language and extract declarations with their spans… 2. Chunk at declaration boundaries, keeping the signature, docstring and body together. 3. Store path, language, symbol name and enclosing scope as metadata for every chunk. 4. Index identifiers in a lexical field as well as a dense field, since exact symbol names a… 5. Include call sites and tests as separate chunk types so a query can ask for usage rather … 6. Re-index on commit or on a schedule, and drop chunks for deleted files to avoid stale def…
- Parse each file with a grammar for its language and extract declarations with their spans and parent scopes.
- Chunk at declaration boundaries, keeping the signature, docstring and body together.
- Store path, language, symbol name and enclosing scope as metadata for every chunk.
- Index identifiers in a lexical field as well as a dense field, since exact symbol names are the common query.
- Include call sites and tests as separate chunk types so a query can ask for usage rather than definition.
- Re-index on commit or on a schedule, and drop chunks for deleted files to avoid stale definitions.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| AST-based chunking | Complete, resolvable units | Per-language parsers and build complexity | The corpus mixes many languages |
| Line-window chunking | Language-agnostic and simple | Splits functions and loses signatures | Files contain long functions |
| Lexical symbol search | Exact identifier matching | No semantic recall | Users describe behaviour rather than naming a symbol |
| Dense code embeddings | Finds similar implementations | Weak on exact names and rare APIs | The query is a symbol or an error message |
Good for
Complete, resolvable units
Cost
Per-language parsers and build complexity
Fails when
The corpus mixes many languages
- The corpus mixes many languages
- Files contain long functions
- Users describe behaviour rather than naming a symbol
- The query is a symbol or an error message
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: AST-based chunking is good for Complete, resolvable units, costs Per-language parsers and build complexity, and fails when The corpus mixes many languages. Line-window chunking is good for Language-agnostic and simple, costs Splits functions and loses signatures, and fails when Files contain long functions. Lexical symbol search is good for Exact identifier matching, costs No semantic recall, and fails when Users describe behaviour rather than naming a symbol. Dense code embeddings is good for Finds similar implementations, costs Weak on exact names and rare APIs, and fails when The query is a symbol or an error message
Failure modes
- Chunk boundary inside a function. The retrieved fragment lacks the signature and cannot be understood or cited. Debug: check that every chunk starts at a declaration and ends at its closing brace.
- Identifier tokenization. A symbol is split into subwords and exact search misses it. Debug: run the analyzer on a known function name and confirm it emits the whole identifier.
- Stale definition. A renamed function still appears because the old chunk was not deleted. Debug: query for the old name and confirm zero hits after re-index.
In production
Track index freshness against the default branch and the share of queries that return a deleted file. Cap chunk size by declaration and fall back to line windows only for files that fail to parse.
Common misconceptions
People think: Code search is just text search over files.
Actually: Structure decides what a useful unit is; a text window cannot guarantee a complete function.
People think: A code-specific embedding model removes the need for exact matching.
Actually: Identifiers and error strings remain exact-match problems, so hybrid retrieval is the norm.
Interview angle
Chunk by syntax, keep symbol metadata, and run lexical and dense retrieval together. Mention call sites and tests as distinct evidence and freshness after renames. Treating repositories as prose documents is the answer that produces half-functions.
Sources
- Feng et al., CodeBERT — pretrained representations for code and natural language.
- Tree-sitter documentation — incremental parsing and syntax trees for declaration-aware chunking.
- Guo et al., GraphCodeBERT — using code structure, not only token sequences, in retrieval models.