Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 20 of 28

Model selection

0 of 28 complete

Why it matters

Teams treat model choice as a one-off quality question, then discover that the best-scoring model is too slow for the interactive path or too expensive at their volume. Selection is a constrained optimisation over several axes, and the constraints come from the product: a 300 ms budget rules out models that win on quality. Without a task-specific evaluation, there is no way to know what the constraint costs.

Mental model

Model selection is hiring for a role: the strongest candidate may not fit the schedule or the budget. The comparison breaks because you can hire two models for different shifts, and swapping one out later costs an integration rather than a conversation.

How it works

Mechanism, step by stepInteractive
  1. 1Split the product into model roles and write the requirement for each: accuracy floor, latency ceiling, context size and output format.
  2. 2Build a labelled evaluation set from real traffic, stratified by the cases that matter, with a fixed scoring rule.
  3. 3Shortlist three candidates per role, using public information only as a filter.
  4. 4Run all candidates on the same set with the same prompts and record quality, p50 and p95 latency, and cost per request.
  5. 5Choose the cheapest model that clears the accuracy floor, not the most accurate model overall.
  6. 6Record the decision, its date and the evidence, and schedule a re-test before the next deprecation.

Step 1 of 6

Split the product into model roles and write the requirement for each: accuracy floor, latency ceiling, context size and output format.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Split the product into model roles and write the requirement for each: accuracy floor, la… 2. Build a labelled evaluation set from real traffic, stratified by the cases that matter, w… 3. Shortlist three candidates per role, using public information only as a filter. 4. Run all candidates on the same set with the same prompts and record quality, p50 and p95 … 5. Choose the cheapest model that clears the accuracy floor, not the most accurate model ove… 6. Record the decision, its date and the evidence, and schedule a re-test before the next de…

  1. Split the product into model roles and write the requirement for each: accuracy floor, latency ceiling, context size and output format.
  2. Build a labelled evaluation set from real traffic, stratified by the cases that matter, with a fixed scoring rule.
  3. Shortlist three candidates per role, using public information only as a filter.
  4. Run all candidates on the same set with the same prompts and record quality, p50 and p95 latency, and cost per request.
  5. Choose the cheapest model that clears the accuracy floor, not the most accurate model overall.
  6. Record the decision, its date and the evidence, and schedule a re-test before the next deprecation.

Trade-offs

ChoiceGood forCostFails when
Most accurate modelBest quality on hard casesHighest cost and latencyTraffic is high volume and answers are simple
Cheapest adequate modelLow cost per requestNarrower margin on edge casesInputs drift outside the evaluated distribution
Small model plus retrievalGood factuality at low costRetrieval pipeline to build and monitorThe task needs reasoning over the retrieved facts
Two-model cascadeCheap common path, strong fallbackRouting logic and two evaluationsYou cannot detect when to escalate
Trade-off mapInteractive

Good for

Best quality on hard cases

Cost

Highest cost and latency

Fails when

Traffic is high volume and answers are simple

  1. Traffic is high volume and answers are simple
  2. Inputs drift outside the evaluated distribution
  3. The task needs reasoning over the retrieved facts
  4. You cannot detect when to escalate

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Most accurate model is good for Best quality on hard cases, costs Highest cost and latency, and fails when Traffic is high volume and answers are simple. Cheapest adequate model is good for Low cost per request, costs Narrower margin on edge cases, and fails when Inputs drift outside the evaluated distribution. Small model plus retrieval is good for Good factuality at low cost, costs Retrieval pipeline to build and monitor, and fails when The task needs reasoning over the retrieved facts. Two-model cascade is good for Cheap common path, strong fallback, costs Routing logic and two evaluations, and fails when You cannot detect when to escalate

Failure modes

  • Benchmark transfer. A leaderboard winner underperforms on your domain. Debug: compare its score on your labelled set with the shortlist baseline before committing.
  • Average hides the tail. Overall accuracy is fine but one critical slice fails. Debug: report scores per slice and gate on the worst slice, not the mean.
  • Stale choice. The model was right at selection time and the task has since shifted. Debug: re-run the evaluation quarterly and diff per-slice scores.

In production

Keep the evaluation harness runnable in one command and store results per model version. Alert when production quality diverges from the offline score, which usually means the input distribution moved.

Common misconceptions

People think: There is one best model.

Actually: There is a best model for a role under a set of constraints, and it changes when the constraints do.

People think: Selection ends at launch.

Actually: Deprecations, price changes and task drift mean the decision needs a review date.

Interview angle

Set up selection as a constrained decision: role requirements, a labelled set, and a rule such as cheapest model above the accuracy floor. Strong answers mention per-slice gating and the review date. Weak answers name a favourite model and justify it with a public benchmark.

Sources