Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 10 of 37

Routing

0 of 37 complete

Why it matters

One model and one prompt for every request is either too expensive or too weak. Routing splits traffic so each class gets an appropriate handler, but routing errors are invisible in aggregate: the system still answers, just with the wrong tool. You need labelled traffic per class, a cost for each confusion, and a way to audit decisions after the fact.

Mental model

Routing is triage at a hospital entrance, not a diagnosis. The metaphor breaks because a triage nurse can reassess a patient who deteriorates, while a router decides once and the chosen handler rarely has authority to send the request back.

How it works

Cost per 1,000 requests · six routing policiesInteractive
Model spendRework from silent misroutesRow total (spend + rework)

Sorted by the row total, cheapest first. Within a row, model spend is the top bar, rework the middle one and their total the bottom one — the quantity the dashed rule compares.

Hover a row for its exact values. Every bar prints its number, and the values table lists all of them.

Cheapest correct policy
$4.64 / 1k
Today's setting
$7.50 / 1k
One silent misroute
$0.25
Step 1/4 · Where the spend sits

Model spend for the best policy is $2.39 per 1,000 against $7.50 for the largest model. On call price alone every routed policy looks like a saving; the row total is $4.64, which is what the rule and the sort compare.

Values
Printed values for every category, in the drawn order. The axis runs from 0 to 18.
categoryModel spendRework from silent misroutesRow total (spend + rework)
Classifier, escalate below 0.75$2.39$2.25$4.64
Keyword rules with a human fallback$3.02$4.05$7.07
Every request to the largest model$7.50$0.00$7.50
Classifier, escalate below 0.60$3.50$5.40$8.90
Classifier, escalate below 0.90$1.43$8.10$9.53
Every request to the cheapest model$0.60$15.75$16.35

Reference rule: today's setting largest model · $7.50 at $7.50.

Worked example, not a measurement. Assumptions: 1,000 requests, 18% of them need the strong tier; a cheap call costs $0.0006, a strong call $0.0075, and each silent misroute costs $0.25 in rework. The chart shows the consequence of the ratio between those prices, not any provider's price list.

Figure. Six routing policies, sorted by total cost per 1,000 requests. Each row is model spend (accent), rework from silent misroutes (warning) and their sum (teal); the dashed rule is today's setting — every request to the largest model at $7.50 — drawn against the same total. This is a worked example from the assumptions in the footnote, not a measurement.

What to notice. Rework, not call price, decides the order. The policy with the cheapest calls spends $0.60 on them and $15.75 on rework, because one silent misroute costs about as much as 400 cheap calls. Lowering the confidence threshold escalates more and misses fewer; raising it saves calls and sends hard requests to the cheap tier, where a wrong answer is plausibly formatted and nobody notices. The two policies under the line win by different margins — one by nearly three dollars per thousand, the other by less than one — which is the point: routing is a decision-cost problem, not a token-price problem.

Text version

A horizontal bar chart titled cost per 1,000 requests, six routing policies, sorted by total ascending. Each row draws three bars — model spend, rework from silent misroutes, and the row total — and every bar prints its value in dollars. A dashed vertical rule sits at $7.50, labelled current setting, largest model, and it is drawn against the row totals. The best policy, classifier escalate below 0.75, spends $2.39 on calls and $2.25 on rework, total $4.64. Keyword rules with a human fallback spends $3.02 and $4.05, total $7.07. Every request to the largest model spends $7.50 with no rework, total $7.50, landing exactly on the rule. Classifier 0.60 spends $3.50 plus $5.40, total $8.90. Classifier 0.90 spends $1.43 plus $8.10, total $9.53. Cheap only spends $0.60 plus $15.75, total $16.35. The card below the chart reports the cheapest correct policy, the current setting, and the cost of one re-contact. The step buttons walk: where the spend sits, why rework dominates, the threshold from both sides, and which totals beat the line.

  1. Enumerate the routes and the observable signal that separates them: intent, length, tenant, required tool.
  2. Start with deterministic rules for the classes you can name and measure how much traffic they cover.
  3. Add a classifier only for the residual, built on labelled examples from your own logs.
  4. Assign a cost to each kind of confusion and tune the confidence threshold against it.
  5. Set a confidence threshold below which the request escalates to a general handler or a human.
  6. Log route, confidence, features and downstream outcome on one record.

Trade-offs

ChoiceGood forCostFails when
Keyword rulesTransparent, instant, freeBrittle to phrasing and languageUsers phrase requests unpredictably
Small classifierCheap and fast at volumeNeeds labels and retrainingClasses shift with the product
Model as routerHandles novel phrasing and many classesAdds a call and latencyThe routing decision is itself unreliable
Always the largest modelNo routing errorsCost and latency on every requestTraffic is dominated by simple requests
Trade-off mapInteractive

Good for

Transparent, instant, free

Cost

Brittle to phrasing and language

Fails when

Users phrase requests unpredictably

  1. Users phrase requests unpredictably
  2. Classes shift with the product
  3. The routing decision is itself unreliable
  4. Traffic is dominated by simple requests

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Keyword rules is good for Transparent, instant, free, costs Brittle to phrasing and language, and fails when Users phrase requests unpredictably. Small classifier is good for Cheap and fast at volume, costs Needs labels and retraining, and fails when Classes shift with the product. Model as router is good for Handles novel phrasing and many classes, costs Adds a call and latency, and fails when The routing decision is itself unreliable. Always the largest model is good for No routing errors, costs Cost and latency on every request, and fails when Traffic is dominated by simple requests

Failure modes

  • Silent misroute. The wrong handler answers plausibly and nobody notices. Debug: sample routed traffic weekly and label the correct route by hand.
  • Stale classes. A new intent falls into the nearest old class with high confidence. Debug: monitor the distribution of confidence and route labels for drift.
  • No escape hatch. A request the router cannot serve loops between handlers. Debug: count requests that reach the fallback and inspect the top unmatched phrasings.

In production

Track route distribution, the confidence histogram and downstream success per route. Alert on a shift in distribution, which usually indicates a product change rather than a model change.

Common misconceptions

People think: A better router removes the need for a fallback.

Actually: Every classifier meets inputs outside its training distribution; the fallback is what makes that safe.

People think: Routing is a model-quality problem.

Actually: It is a decision-cost problem: the right route depends on what a mistake costs in each direction.

Interview angle

Strong answers define the classes, the signal, the cost matrix and the fallback before choosing a mechanism, and describe how misroutes are detected after launch. Weak answers propose an LLM router for everything and never mention what happens to a request that fits no class.

Sources