Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 29 of 38

Multi-provider routing

0 of 38 complete

Why it matters

A single provider is a single point of failure and a single price. Routing across providers can cut cost on easy traffic, keep latency inside a target when one provider throttles, and provide failover without a separate mechanism. But each route is a different model with different instruction-following, output format and token accounting, so the router has to be evaluated per route and must record which route produced which answer.

Mental model

It is a train operator choosing between lines for each journey: the destination is the same, the route differs in time and cost. The analogy breaks because passengers do not change language mid-journey, while two models can answer the same request in incompatible formats.

How it works

Multi-provider routing topologyInteractive

Routing chooses on every request; failover switches only when a route is unhealthy. The two are not the same policy.

Step 1/6 · Classify the request
RequestControlRoutes + providersValidate

Step 1 of 6

Classify the request

Task type, difficulty, latency class and tenant tier are estimated before a route is chosen. The classifier can abstain; an abstention uses the default route.

service

Control

Task classifier

difficulty · type

Estimates difficulty and task type. If the classifier cannot separate easy from hard traffic, cost-based routing becomes a random discount on quality.

2 connections

syncasyncreturnfailurehighlighted path = current step4 zones · 13 nodes · 18 edges · illustrative geometry

Figure. The topology guarantees that every request is classified, routed by an explicit policy with health input, dispatched to a named provider, validated against the expected schema, and served with the chosen provider recorded in metadata. It does not guarantee quality parity between routes, that the classifier is right, or that a cache hit came from the same route: those require per-route evaluation, share caps and provider-keyed cache entries.

What to notice. Routing and failover meet at the router but are different policies: routing decides every request, failover reacts to an unhealthy route. The red edges are the failures that hide, a cheap route returning prose instead of an object and a cross-provider cache hit served as if it came from the selected route.

Text version

A multi-provider routing topology with four zones. A request reaches a router in the control lane, which asks a task classifier for difficulty and type, reads a routing policy for the default, overrides and share caps, and reads a health and latency signal. Easy traffic goes to the cheap route and is dispatched to provider A; hard or schema-critical traffic goes to the frontier route and is dispatched to provider B. Both provider outputs pass one schema validator, which can retry a schema failure on the frontier route and writes validated answers to a provider-keyed response cache before returning them with route metadata. Route decisions and outcomes feed a telemetry store back into policy with hysteresis. Red dashed failure edges show prose instead of JSON from the cheap route and a cross-provider cache hit. Six steps cover classification, policy and health, route and provider choice, schema validation, provenance, and feedback.

  1. Define the routing dimensions: task type, difficulty, latency class, tenant tier and current provider health.
  2. Keep one internal request and response schema, and adapt each provider behind it.
  3. Set a default route and explicit overrides, so an unclassified request does not land on the cheapest model by accident.
  4. Evaluate every route on the same task set, including structured-output validity and refusal behaviour.
  5. Record the chosen provider and model in the response metadata and trace attributes.
  6. Feed latency, error rate and cost back into the policy, with hysteresis so routes do not oscillate.

Trade-offs

ChoiceGood forCostFails when
Cost-based routingLower spend on easy trafficQuality variance across the productThe classifier cannot tell difficulty
Latency-based routingProtects interactive pathsOscillation as providers varyProvider latencies are similar
Health-based failoverAvailability during throttlingFailover capacity must be realThe backup quota is small
Single default with overridesSimple and predictableLess cost optimisationTraffic is dominated by one easy task type
Trade-off mapInteractive

Good for

Lower spend on easy traffic

Cost

Quality variance across the product

Fails when

The classifier cannot tell difficulty

  1. The classifier cannot tell difficulty
  2. Provider latencies are similar
  3. The backup quota is small
  4. Traffic is dominated by one easy task type

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Cost-based routing is good for Lower spend on easy traffic, costs Quality variance across the product, and fails when The classifier cannot tell difficulty. Latency-based routing is good for Protects interactive paths, costs Oscillation as providers vary, and fails when Provider latencies are similar. Health-based failover is good for Availability during throttling, costs Failover capacity must be real, and fails when The backup quota is small. Single default with overrides is good for Simple and predictable, costs Less cost optimisation, and fails when Traffic is dominated by one easy task type

Failure modes

  • Route without schema validation. A cheaper model returns prose where a structured object was required and the parser fails downstream. Debug: measure structured-output validity per route before enabling it.
  • Cache keyed without provider. A cached answer from one model is served as if it came from another. Debug: include provider and model in the cache key and assert the key changes when the route changes.
  • Route oscillation. Two providers alternate as the cheapest or fastest and the policy flips every minute. Debug: chart route share over time and look for rapid alternation.

In production

Report quality, latency and cost per route, and treat a new route as a new model that needs its own evaluation and canary. Cap the share any unproven route can take.

Common misconceptions

People think: Multi-provider routing is the same as failover.

Actually: Failover switches when the primary is unhealthy; routing chooses on every request, which means every route must be good enough.

People think: Routing by price reduces cost per task.

Actually: Only if the cheaper route still completes the task; retries and corrections can make it more expensive.

Interview angle

Name the routing dimensions, the evaluation per route and the metadata that records the decision. Explain why health-based failover and per-request routing differ. A fallback list described as routing is the weak answer.

Sources