Why it matters
A private corpus is full of strings whose exact form carries the meaning: SKUs, invoice prefixes, API error codes, legal citations. Dense retrieval blurs those strings into a vector and can rank a semantically similar passage above the one containing the exact identifier. Lexical search exists to guarantee that a token present in the query is honoured, and its analyzer configuration is the part teams most often leave at a default that silently mangles their identifiers.
Mental model
Lexical search is a concordance at the back of a book: you look up the word and get every page that prints it. The concordance is exact and fast, and it is also useless when the author used a synonym you did not think to look up.
How it works
Synthetic analyzer record · query ERR-2041
One query, one field, three analysis settings
bounding boxes
Step 1Fix the field first
The query is a code and the field holds a code. Before any scoring, decide that the code must survive analysis as a single token, because an ordinary text analyzer splits it at the hyphen.
LEXICAL SEARCH · ANALYZER RECORD · QUERY ERR-2041
What the analyzer emits
| Stage | Setting |
|---|---|
| normalise | NFKC |
| tokenize | standard |
| lowercase | yes |
| stemming | none |
| code field | keyword |
| Chain | Tokens | Code |
|---|---|---|
| standard | 13 | split |
| +stemming | 13 | split |
| keyword | 1 | intact |
| Query token | Matches | Kind |
|---|---|---|
| err-2041 | ERR-2041 | exact |
| err | 7 documents | ambiguous |
| 2041 | 7 documents | ambiguous |
13 tokens · 1 code intact · exact 1 · split 7 · 6 false matches avoided
Step record
- Query
- ERR-2041
- Fields
- text + code
- Expected
- 1 document
The text version summarises this figure; select a region for its full text.
Figure. One field and one query through three analysis settings: the standard chain emits 13 tokens and splits the code into err and 2041, and enabling stemming changes neither. A keyword field on the same value emits one token, the code intact. The keyword field matches 1 document; the split tokens match 7, six of them false.
What to notice. Synthetic field and query. Token counts are exact for the sentence shown; the seven matching documents are an authored count, not a measured one. The figure shows what analysis decides before scoring, not a relevance result.
Text version
An analyzer record for the query ERR-2041 against the field text “Invoice NW-8841 · ERR-2041 on checkout. Refunds are processed within 14 days.” The analysis chain runs normalise NFKC, tokenize standard, lowercase yes, stemming none, with a keyword field for the code. The standard setting emits 13 tokens with the code split; enabling stemming emits the same 13 with the code still split; the keyword field emits 1 token, the code intact. A match table shows the keyword query matching ERR-2041 exactly and the split tokens err and 2041 each matching seven documents, six of them false. Steps walk the field choice, the chain, the token counts, the match, the ranking, and an index-query analyzer mismatch.
- List the tokens that must match exactly, such as identifiers, codes and names, and choose an analyzer that preserves them as single terms.
- Define the analysis chain explicitly: Unicode normalisation, tokenizer, lowercase filter, then any stemming, with a keyword field for exact values.
- Index both an analysed text field for prose and an unanalysed keyword field for identifiers, rather than compromising on one.
- Use phrase and proximity queries where word order changes meaning, and boolean operators where a term is mandatory.
- Score with a term-weighting function such as BM25 and inspect the per-term contribution for a failing query.
- Test with the corpus vocabulary, including hyphenated words, product codes and non-English diacritics, before tuning relevance.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Stemmed text field | Matches inflections and plurals | Conflates distinct terms and loses precision | Identifiers or names differ only by suffix |
| Keyword-only field | Exact recall on codes and titles | No partial or fuzzy matching | Users type fragments or misspellings |
| Strict boolean query | Guarantees required terms | Zero results when one term is absent | Users add filler words to a precise query |
| Fuzzy matching | Typo tolerance | Expands the candidate set with near-misses | The corpus contains many similar codes |
Good for
Matches inflections and plurals
Cost
Conflates distinct terms and loses precision
Fails when
Identifiers or names differ only by suffix
- Identifiers or names differ only by suffix
- Users type fragments or misspellings
- Users add filler words to a precise query
- The corpus contains many similar codes
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Stemmed text field is good for Matches inflections and plurals, costs Conflates distinct terms and loses precision, and fails when Identifiers or names differ only by suffix. Keyword-only field is good for Exact recall on codes and titles, costs No partial or fuzzy matching, and fails when Users type fragments or misspellings. Strict boolean query is good for Guarantees required terms, costs Zero results when one term is absent, and fails when Users add filler words to a precise query. Fuzzy matching is good for Typo tolerance, costs Expands the candidate set with near-misses, and fails when The corpus contains many similar codes
Failure modes
- Analyzer mismatch between index and query. The document was indexed with stemming and queried without it, so a plural never matches. Debug: run the analyzer on a sample query and on the document field, and compare the emitted token lists.
- Identifier destruction. A hyphenated part number is split into three tokens and matches unrelated parts. Debug: fetch the stored terms for one known document and check the identifier survived as one token.
- Stopword removal of meaningful words. Removing common words makes a phrase query match a different sentence. Debug: disable stopword filtering for the field and compare the top ten results.
In production
Log the analysed form of queries that return nothing and review the list weekly; analyzer bugs surface as clusters of zero-result searches. Pin the analyzer configuration with the index mapping so a rebuild reproduces the same terms.
Common misconceptions
People think: Lexical search is obsolete once you have embeddings.
Actually: Exact identifiers and rare terms are precisely where dense vectors lose, which is why production stacks run both.
People think: Stemming always improves recall.
Actually: It helps prose and harms codes, names and short fields where a suffix is part of the identity.
Interview angle
Open with the tokens that must match literally, then walk the analyzer chain and where it mangles identifiers. A convincing answer contrasts vocabulary mismatch with the exact-match guarantee and reaches for hybrid retrieval. Answers that call lexical search legacy miss the part numbers, error codes and surnames that dense vectors routinely lose.
Sources
- PostgreSQL, Full Text Search — analyzers, dictionaries and term matching in a relational engine.
- Elasticsearch, Similarity module — how term-weighting is configured in a production lexical engine.
- Manning, Raghavan & Schütze, Introduction to Information Retrieval — tokenization, stemming and the retrieval models behind lexical search.