Skip to content
AI Engineer Guide.
midReportedcurated

Pasting an order ID like ORD-88213 returns generic refund-policy chunks after a move from keyword to vector search. Diagnose and fix.

30-second answer

Likely follow-ups

Open a question to rehearse the answer.

How do you verify BM25 is still contributing after fusion, rather than being drowned by the dense list?
  • Track per-retriever contribution to the final top-k and per-slice recall on identifier queries, not the aggregate.
  • Log which candidates each retriever returned for sampled ID queries and check the BM25-only hits still reach the prompt.
  • A rise in overlap between the two lists usually means the analyser changed and lexical search is now duplicating dense behaviour.
Which side do you normalise, the query, the index, or both, and what breaks if they disagree?
  • Run both sides through one analyser configuration, because query-side and index-side tokenisation that disagree produce silent misses.
  • Lower-case and strip punctuation in the preprocessor, and keep the raw value in a keyword field for exact-match lookups.
  • Test five known error codes through the analyser: if two produce the same token stream, the index cannot separate them.
Would a metadata exact-match lookup beat adding a lexical index for order IDs, and when?
  • An exact-match lookup on a stored field is faster and simpler when the identifier is already a column you keep.
  • It stops working when identifiers appear in body text, are mistyped, or need partial matching, which is where a lexical index earns its cost.
  • Most stacks end up with both: exact lookup for known fields, BM25 for everything else, fused only where the two can disagree.

Red flags

  • lower threshold
  • blame the LLM

Concepts this tests

Tags

Source: forwarddeployedprep.com

More in RAG & retrieval pipelines