Why it matters
A single retrieval path forces every query through the same index with the same parameters. That is wasteful for chit-chat and wrong for questions that need a database aggregate or a code search. Routing adds a cheap decision in front of retrieval so each query type gets an appropriate handler, but the decision itself can be wrong, and a misroute is invisible unless the route is logged with the outcome.
Mental model
Routing is a hospital triage desk: each arrival is sent to the department that can treat it. The metaphor breaks when a patient has symptoms from two departments, which is why the router needs a fallback rather than a single label.
How it works
Sorted by service time, ascending: the claim is what each handler costs, not the order requests arrive in.
Hover a row for its exact values. Every bar prints its number, and the values table lists all of them.
- Dense path
- 420 ms
- No retrieval
- 8 ms
- Ratio
- about 50×
Every request pays 420 ms: 35 to embed, 45 for the ANN search, 300 to rerank and 40 to assemble the prompt. A policy question needs all four steps.
Values
| category | Service time |
|---|---|
| No retrieval · greetings and chit-chat · no index touched | 8 ms |
| Lexical index · identifier and error-code lookups · exact term match | 45 ms |
| Structured query · counts, aggregates, date ranges · generates and runs SQL | 130 ms |
| Tool call · code search, calculator, ticketing · one tool round trip | 240 ms |
| Dense retrieval · policy and prose questions · 35 embed + 45 ANN + 300 rerank + 40 assembly | 420 ms |
Reference rule: one dense path for all · 420 ms at 420 ms.
Illustrative service times, not a benchmark. Components: embedding 35 ms, ANN search 45 ms, reranking 300 ms, prompt assembly 40 ms; lexical lookup 45 ms, structured query 130 ms, one tool round trip 240 ms, no retrieval 8 ms. Real numbers move with index size, model, hardware and cache hit rate.
Figure. Service time for one request on each of the five query handlers, sorted from cheapest to most expensive. The dashed rule is the dense path used for everything today, at 420 ms. Times are built from the stated components in the footnote: an illustrative worked example, not a benchmark of any index or model.
What to notice. A greeting does not need evidence, and today it pays 420 ms for retrieval it cannot use. The dense path stays the only handler that can answer a policy question, so the claim is not that it is wasteful — it is that using it for every class pays about 50 times the no-retrieval cost on traffic that needed no retrieval, and the decision has to be made before generation from features the router can see. The figure omits reranker quality, index size, cache hit rate and the cost of a misroute; route errors appear on the agent-routing page.
Text version
A horizontal bar chart of service time in milliseconds for five query routes, sorted ascending. Every bar prints its time and a dashed rule at 420 ms marks the dense path used for everything today. No retrieval is 8 ms, lexical is 45 ms, structured is 130 ms, tool is 240 ms and dense is 420 ms, the last bar ending on the rule. The card reports the dense path at 420 ms, the no-retrieval path at 8 ms and a ratio of about 50 times. Steps highlight the dense bar and its four components in the note, then the no-retrieval class, then the three cheap handlers, then the whole chart.
- Enumerate the handlers you actually have: lexical, dense, structured, tool and no-retrieval.
- Write the routing decision as explicit rules first, using query features such as identifier patterns, question words and length.
- Add a classifier only for the cases rules cannot separate, trained on labelled examples of each route.
- Calibrate the classifier's confidence and send low-confidence queries to a broad default handler.
- Log the chosen route, the confidence and the downstream outcome for every request.
- Evaluate routing accuracy separately from answer quality, so a good answer from the wrong route is still visible.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Rule-based routing | Transparent and instant | Brittle on unanticipated phrasing | Users phrase requests in many ways |
| Classifier routing | Handles free-form queries | Training data and calibration work | Traffic distribution shifts |
| Route to one index only | Simple and cacheable | Misses queries the other index would serve | The corpus mixes prose and identifiers |
| Retrieve for everything | No misroute risk | Latency and noise on queries that need no evidence | Traffic includes chit-chat or commands |
Good for
Transparent and instant
Cost
Brittle on unanticipated phrasing
Fails when
Users phrase requests in many ways
- Users phrase requests in many ways
- Traffic distribution shifts
- The corpus mixes prose and identifiers
- Traffic includes chit-chat or commands
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Rule-based routing is good for Transparent and instant, costs Brittle on unanticipated phrasing, and fails when Users phrase requests in many ways. Classifier routing is good for Handles free-form queries, costs Training data and calibration work, and fails when Traffic distribution shifts. Route to one index only is good for Simple and cacheable, costs Misses queries the other index would serve, and fails when The corpus mixes prose and identifiers. Retrieve for everything is good for No misroute risk, costs Latency and noise on queries that need no evidence, and fails when Traffic includes chit-chat or commands
Failure modes
- Silent misroute. A structured question goes to the text index and gets a vague answer that looks acceptable. Debug: join route labels with answer quality on a labelled sample and inspect the worst cell.
- Overconfident classifier. Every query is routed with high confidence, including out-of-distribution ones. Debug: plot confidence against accuracy and check the calibration curve on held-out traffic.
- Missing default path. An unclassified query raises an error instead of falling back. Debug: send an empty and a gibberish query through the router and confirm both reach a handler.
In production
Track route distribution, confidence histogram and per-route answer quality. Alert on a sudden shift in distribution, which usually means a client changed its query format.
Common misconceptions
People think: Routing is only about cost.
Actually: It is also about correctness: some questions cannot be answered by the text index at all.
People think: A good classifier removes the need for rules.
Actually: Deterministic features such as identifier patterns are cheaper and more reliable than a model for the cases they cover.
Interview angle
List the handlers, start with rules, and insist on calibration and a default path. Evaluate routing separately from answering. Describing an intent classifier without saying what happens when it is wrong or unsure is the incomplete answer.
Sources
- Anthropic, Building effective agents — routing as a workflow pattern with a classifier and specialised branches.
- Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey — adaptive retrieval, including deciding whether to retrieve at all.
- Asai et al., Self-RAG — learning when retrieval is needed via reflection tokens.