Why it matters
One model and one prompt for every request is either too expensive or too weak. Routing splits traffic so each class gets an appropriate handler, but routing errors are invisible in aggregate: the system still answers, just with the wrong tool. You need labelled traffic per class, a cost for each confusion, and a way to audit decisions after the fact.
Mental model
Routing is triage at a hospital entrance, not a diagnosis. The metaphor breaks because a triage nurse can reassess a patient who deteriorates, while a router decides once and the chosen handler rarely has authority to send the request back.
How it works
Sorted by the row total, cheapest first. Within a row, model spend is the top bar, rework the middle one and their total the bottom one — the quantity the dashed rule compares.
Hover a row for its exact values. Every bar prints its number, and the values table lists all of them.
- Cheapest correct policy
- $4.64 / 1k
- Today's setting
- $7.50 / 1k
- One silent misroute
- $0.25
Model spend for the best policy is $2.39 per 1,000 against $7.50 for the largest model. On call price alone every routed policy looks like a saving; the row total is $4.64, which is what the rule and the sort compare.
Values
| category | Model spend | Rework from silent misroutes | Row total (spend + rework) |
|---|---|---|---|
| Classifier, escalate below 0.75 | $2.39 | $2.25 | $4.64 |
| Keyword rules with a human fallback | $3.02 | $4.05 | $7.07 |
| Every request to the largest model | $7.50 | $0.00 | $7.50 |
| Classifier, escalate below 0.60 | $3.50 | $5.40 | $8.90 |
| Classifier, escalate below 0.90 | $1.43 | $8.10 | $9.53 |
| Every request to the cheapest model | $0.60 | $15.75 | $16.35 |
Reference rule: today's setting largest model · $7.50 at $7.50.
Worked example, not a measurement. Assumptions: 1,000 requests, 18% of them need the strong tier; a cheap call costs $0.0006, a strong call $0.0075, and each silent misroute costs $0.25 in rework. The chart shows the consequence of the ratio between those prices, not any provider's price list.
Figure. Six routing policies, sorted by total cost per 1,000 requests. Each row is model spend (accent), rework from silent misroutes (warning) and their sum (teal); the dashed rule is today's setting — every request to the largest model at $7.50 — drawn against the same total. This is a worked example from the assumptions in the footnote, not a measurement.
What to notice. Rework, not call price, decides the order. The policy with the cheapest calls spends $0.60 on them and $15.75 on rework, because one silent misroute costs about as much as 400 cheap calls. Lowering the confidence threshold escalates more and misses fewer; raising it saves calls and sends hard requests to the cheap tier, where a wrong answer is plausibly formatted and nobody notices. The two policies under the line win by different margins — one by nearly three dollars per thousand, the other by less than one — which is the point: routing is a decision-cost problem, not a token-price problem.
Text version
A horizontal bar chart titled cost per 1,000 requests, six routing policies, sorted by total ascending. Each row draws three bars — model spend, rework from silent misroutes, and the row total — and every bar prints its value in dollars. A dashed vertical rule sits at $7.50, labelled current setting, largest model, and it is drawn against the row totals. The best policy, classifier escalate below 0.75, spends $2.39 on calls and $2.25 on rework, total $4.64. Keyword rules with a human fallback spends $3.02 and $4.05, total $7.07. Every request to the largest model spends $7.50 with no rework, total $7.50, landing exactly on the rule. Classifier 0.60 spends $3.50 plus $5.40, total $8.90. Classifier 0.90 spends $1.43 plus $8.10, total $9.53. Cheap only spends $0.60 plus $15.75, total $16.35. The card below the chart reports the cheapest correct policy, the current setting, and the cost of one re-contact. The step buttons walk: where the spend sits, why rework dominates, the threshold from both sides, and which totals beat the line.
- Enumerate the routes and the observable signal that separates them: intent, length, tenant, required tool.
- Start with deterministic rules for the classes you can name and measure how much traffic they cover.
- Add a classifier only for the residual, built on labelled examples from your own logs.
- Assign a cost to each kind of confusion and tune the confidence threshold against it.
- Set a confidence threshold below which the request escalates to a general handler or a human.
- Log route, confidence, features and downstream outcome on one record.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Keyword rules | Transparent, instant, free | Brittle to phrasing and language | Users phrase requests unpredictably |
| Small classifier | Cheap and fast at volume | Needs labels and retraining | Classes shift with the product |
| Model as router | Handles novel phrasing and many classes | Adds a call and latency | The routing decision is itself unreliable |
| Always the largest model | No routing errors | Cost and latency on every request | Traffic is dominated by simple requests |
Good for
Transparent, instant, free
Cost
Brittle to phrasing and language
Fails when
Users phrase requests unpredictably
- Users phrase requests unpredictably
- Classes shift with the product
- The routing decision is itself unreliable
- Traffic is dominated by simple requests
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Keyword rules is good for Transparent, instant, free, costs Brittle to phrasing and language, and fails when Users phrase requests unpredictably. Small classifier is good for Cheap and fast at volume, costs Needs labels and retraining, and fails when Classes shift with the product. Model as router is good for Handles novel phrasing and many classes, costs Adds a call and latency, and fails when The routing decision is itself unreliable. Always the largest model is good for No routing errors, costs Cost and latency on every request, and fails when Traffic is dominated by simple requests
Failure modes
- Silent misroute. The wrong handler answers plausibly and nobody notices. Debug: sample routed traffic weekly and label the correct route by hand.
- Stale classes. A new intent falls into the nearest old class with high confidence. Debug: monitor the distribution of confidence and route labels for drift.
- No escape hatch. A request the router cannot serve loops between handlers. Debug: count requests that reach the fallback and inspect the top unmatched phrasings.
In production
Track route distribution, the confidence histogram and downstream success per route. Alert on a shift in distribution, which usually indicates a product change rather than a model change.
Common misconceptions
People think: A better router removes the need for a fallback.
Actually: Every classifier meets inputs outside its training distribution; the fallback is what makes that safe.
People think: Routing is a model-quality problem.
Actually: It is a decision-cost problem: the right route depends on what a mistake costs in each direction.
Interview angle
Strong answers define the classes, the signal, the cost matrix and the fallback before choosing a mechanism, and describe how misroutes are detected after launch. Weak answers propose an LLM router for everything and never mention what happens to a request that fits no class.
Sources
- Anthropic, Building effective agents — routing as a workflow pattern with specialised downstream prompts.
- Chen, Zaharia and Zou, FrugalGPT — cascades and routing across models to trade cost against quality.