midReportedcurated
Pasting an order ID like ORD-88213 returns generic refund-policy chunks after a move from keyword to vector search. Diagnose and fix.
30-second answer
- Rare identifiers carry almost no semantic signal, so the ID chunk is not near the query vector and dense search returns the nearest topical match instead.
- Confirm by printing the dense ranking and the chunk's rank, because this failure looks identical to a document that is missing from the corpus.
- Add BM25 alongside dense and fuse with RRF, then check tokenisation, since a punctuated or lower-cased copy of the ID will not match without a normalising analyser.
- Add exact-ID questions to the golden set so the next change cannot silently drop them again.
- Hybrid doubles index and query work, which is cheap next to a wrong support answer, but it does not rescue an analyser that collapses distinct codes into one token.
Likely follow-ups
Open a question to rehearse the answer.
How do you verify BM25 is still contributing after fusion, rather than being drowned by the dense list?
- Track per-retriever contribution to the final top-k and per-slice recall on identifier queries, not the aggregate.
- Log which candidates each retriever returned for sampled ID queries and check the BM25-only hits still reach the prompt.
- A rise in overlap between the two lists usually means the analyser changed and lexical search is now duplicating dense behaviour.
Which side do you normalise, the query, the index, or both, and what breaks if they disagree?
- Run both sides through one analyser configuration, because query-side and index-side tokenisation that disagree produce silent misses.
- Lower-case and strip punctuation in the preprocessor, and keep the raw value in a keyword field for exact-match lookups.
- Test five known error codes through the analyser: if two produce the same token stream, the index cannot separate them.
Would a metadata exact-match lookup beat adding a lexical index for order IDs, and when?
- An exact-match lookup on a stored field is faster and simpler when the identifier is already a column you keep.
- It stops working when identifiers appear in body text, are mistyped, or need partial matching, which is where a lexical index earns its cost.
- Most stacks end up with both: exact lookup for known fields, BM25 for everything else, fused only where the two can disagree.
Red flags
- lower threshold
- blame the LLM
Concepts this tests
Tags
Source: forwarddeployedprep.com
More in RAG & retrieval pipelines
- Explain parent-child (small-to-big) retrieval. What problem does it solve and what does it cost you?
- Explain HNSW. What do M, ef_construction and ef_search each control, and which can you change in production?
- How would you tune HNSW for 10 million vectors when the product needs p95 retrieval under 100 ms?