Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 2 of 57

Lexical search

0 of 57 complete

Why it matters

A private corpus is full of strings whose exact form carries the meaning: SKUs, invoice prefixes, API error codes, legal citations. Dense retrieval blurs those strings into a vector and can rank a semantically similar passage above the one containing the exact identifier. Lexical search exists to guarantee that a token present in the query is honoured, and its analyzer configuration is the part teams most often leave at a default that silently mangles their identifiers.

Mental model

Lexical search is a concordance at the back of a book: you look up the word and get every page that prints it. The concordance is exact and fast, and it is also useless when the author used a synonym you did not think to look up.

How it works

What the analyzer emits for one queryInteractive

Synthetic analyzer record · query ERR-2041

One query, one field, three analysis settings

bounding boxes

Step 1/6 · Fix the field first

Step 1Fix the field first

The query is a code and the field holds a code. Before any scoring, decide that the code must survive analysis as a single token, because an ordinary text analyzer splits it at the hyphen.

LEXICAL SEARCH · ANALYZER RECORD · QUERY ERR-2041

, running header, region 1. Select to inspect.

What the analyzer emits

, title, region 2. Select to inspect.
Field

Invoice NW-8841 · ERR-2041 on checkout. Refunds are processed within 14 days.

, paragraph Field, region 3. Select to inspect.
ANALYSIS CHAIN
StageSetting
normaliseNFKC
tokenizestandard
lowercaseyes
stemmingnone
code fieldkeyword
, table ANALYSIS CHAIN, region 4. Select to inspect.
EMITTED
ChainTokensCode
standard13split
+stemming13split
keyword1intact
, table EMITTED, region 5. Select to inspect.
MATCHES
Query tokenMatchesKind
err-2041ERR-2041exact
err7 documentsambiguous
20417 documentsambiguous
, table MATCHES, region 6. Select to inspect.
Resultexact 1 document · split 7 documents · 6 false matches avoided
, form field Result, region 7. Select to inspect.
Pin it

The chain is pinned with the index mapping. A rebuild that changes the chain changes every term, so index and query must run the same settings.

, paragraph Pin it, region 8. Select to inspect.

13 tokens · 1 code intact · exact 1 · split 7 · 6 false matches avoided

, running footer, region 9. Select to inspect.
9 regions

Step record

Query
ERR-2041
Fields
text + code
Expected
1 document

The text version summarises this figure; select a region for its full text.

Figure. One field and one query through three analysis settings: the standard chain emits 13 tokens and splits the code into err and 2041, and enabling stemming changes neither. A keyword field on the same value emits one token, the code intact. The keyword field matches 1 document; the split tokens match 7, six of them false.

What to notice. Synthetic field and query. Token counts are exact for the sentence shown; the seven matching documents are an authored count, not a measured one. The figure shows what analysis decides before scoring, not a relevance result.

Text version

An analyzer record for the query ERR-2041 against the field text “Invoice NW-8841 · ERR-2041 on checkout. Refunds are processed within 14 days.” The analysis chain runs normalise NFKC, tokenize standard, lowercase yes, stemming none, with a keyword field for the code. The standard setting emits 13 tokens with the code split; enabling stemming emits the same 13 with the code still split; the keyword field emits 1 token, the code intact. A match table shows the keyword query matching ERR-2041 exactly and the split tokens err and 2041 each matching seven documents, six of them false. Steps walk the field choice, the chain, the token counts, the match, the ranking, and an index-query analyzer mismatch.

  1. List the tokens that must match exactly, such as identifiers, codes and names, and choose an analyzer that preserves them as single terms.
  2. Define the analysis chain explicitly: Unicode normalisation, tokenizer, lowercase filter, then any stemming, with a keyword field for exact values.
  3. Index both an analysed text field for prose and an unanalysed keyword field for identifiers, rather than compromising on one.
  4. Use phrase and proximity queries where word order changes meaning, and boolean operators where a term is mandatory.
  5. Score with a term-weighting function such as BM25 and inspect the per-term contribution for a failing query.
  6. Test with the corpus vocabulary, including hyphenated words, product codes and non-English diacritics, before tuning relevance.

Trade-offs

ChoiceGood forCostFails when
Stemmed text fieldMatches inflections and pluralsConflates distinct terms and loses precisionIdentifiers or names differ only by suffix
Keyword-only fieldExact recall on codes and titlesNo partial or fuzzy matchingUsers type fragments or misspellings
Strict boolean queryGuarantees required termsZero results when one term is absentUsers add filler words to a precise query
Fuzzy matchingTypo toleranceExpands the candidate set with near-missesThe corpus contains many similar codes
Trade-off mapInteractive

Good for

Matches inflections and plurals

Cost

Conflates distinct terms and loses precision

Fails when

Identifiers or names differ only by suffix

  1. Identifiers or names differ only by suffix
  2. Users type fragments or misspellings
  3. Users add filler words to a precise query
  4. The corpus contains many similar codes

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Stemmed text field is good for Matches inflections and plurals, costs Conflates distinct terms and loses precision, and fails when Identifiers or names differ only by suffix. Keyword-only field is good for Exact recall on codes and titles, costs No partial or fuzzy matching, and fails when Users type fragments or misspellings. Strict boolean query is good for Guarantees required terms, costs Zero results when one term is absent, and fails when Users add filler words to a precise query. Fuzzy matching is good for Typo tolerance, costs Expands the candidate set with near-misses, and fails when The corpus contains many similar codes

Failure modes

  • Analyzer mismatch between index and query. The document was indexed with stemming and queried without it, so a plural never matches. Debug: run the analyzer on a sample query and on the document field, and compare the emitted token lists.
  • Identifier destruction. A hyphenated part number is split into three tokens and matches unrelated parts. Debug: fetch the stored terms for one known document and check the identifier survived as one token.
  • Stopword removal of meaningful words. Removing common words makes a phrase query match a different sentence. Debug: disable stopword filtering for the field and compare the top ten results.

In production

Log the analysed form of queries that return nothing and review the list weekly; analyzer bugs surface as clusters of zero-result searches. Pin the analyzer configuration with the index mapping so a rebuild reproduces the same terms.

Common misconceptions

People think: Lexical search is obsolete once you have embeddings.

Actually: Exact identifiers and rare terms are precisely where dense vectors lose, which is why production stacks run both.

People think: Stemming always improves recall.

Actually: It helps prose and harms codes, names and short fields where a suffix is part of the identity.

Interview angle

Open with the tokens that must match literally, then walk the analyzer chain and where it mangles identifiers. A convincing answer contrasts vocabulary mismatch with the exact-match guarantee and reaches for hybrid retrieval. Answers that call lexical search legacy miss the part numbers, error codes and surnames that dense vectors routinely lose.

Sources