Why it matters
A team choosing a model faces dozens of names with overlapping claims and no shared measurement. The useful structure is a set of axes: what the model was trained to do, how much context it accepts, which modalities it handles, how it is served and what it costs at your volume. Without those axes, selection becomes fashion, and the choice is revisited every time a release appears.
Mental model
The landscape is a hardware catalogue: categories, sockets and power envelopes matter more than the model number. It breaks because model specs are not stable; a family can change tokeniser or behaviour between releases while keeping its name, so a part that fit last year may not fit now.
How it works
- 1Group candidates by the capability the product needs: text generation, reasoning, code, vision, speech, embeddings or classification.
- 2Within a group, record context limit, modality support, tool-calling support and whether the weights are downloadable.
- 3Filter by deployment: hosted API, managed open-weight endpoint, or self-hosted on hardware you already operate.
- 4Shortlist two or three per role and test them on your own task set with your own prompts.
- 5Note the tokeniser family, since token counts and therefore cost differ for the same text.
- 6Re-check the shortlist on a schedule rather than on every announcement, so churn does not become the roadmap.
Step 1 of 6
Group candidates by the capability the product needs: text generation, reasoning, code, vision, speech, embeddings or classification.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Group candidates by the capability the product needs: text generation, reasoning, code, v… 2. Within a group, record context limit, modality support, tool-calling support and whether … 3. Filter by deployment: hosted API, managed open-weight endpoint, or self-hosted on hardwar… 4. Shortlist two or three per role and test them on your own task set with your own prompts. 5. Note the tokeniser family, since token counts and therefore cost differ for the same text. 6. Re-check the shortlist on a schedule rather than on every announcement, so churn does not…
- Group candidates by the capability the product needs: text generation, reasoning, code, vision, speech, embeddings or classification.
- Within a group, record context limit, modality support, tool-calling support and whether the weights are downloadable.
- Filter by deployment: hosted API, managed open-weight endpoint, or self-hosted on hardware you already operate.
- Shortlist two or three per role and test them on your own task set with your own prompts.
- Note the tokeniser family, since token counts and therefore cost differ for the same text.
- Re-check the shortlist on a schedule rather than on every announcement, so churn does not become the roadmap.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Frontier general model | Broad capability, handles novel tasks | Price and latency at the top of the range | The task is narrow and high volume |
| Small open-weight model | Cheap, controllable, self-hostable | Narrower generalisation and more evaluation work | Inputs are open-ended and adversarial |
| Specialised model per task | Best quality per task | Several integrations and evaluation suites | The product needs one model to do everything |
| Single family for all roles | Shared tooling and tokeniser | Some roles overpay for capability | A role has a hard latency or cost ceiling |
Good for
Broad capability, handles novel tasks
Cost
Price and latency at the top of the range
Fails when
The task is narrow and high volume
- The task is narrow and high volume
- Inputs are open-ended and adversarial
- The product needs one model to do everything
- A role has a hard latency or cost ceiling
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Frontier general model is good for Broad capability, handles novel tasks, costs Price and latency at the top of the range, and fails when The task is narrow and high volume. Small open-weight model is good for Cheap, controllable, self-hostable, costs Narrower generalisation and more evaluation work, and fails when Inputs are open-ended and adversarial. Specialised model per task is good for Best quality per task, costs Several integrations and evaluation suites, and fails when The product needs one model to do everything. Single family for all roles is good for Shared tooling and tokeniser, costs Some roles overpay for capability, and fails when A role has a hard latency or cost ceiling
Failure modes
- Name-based selection. A team picks a model because it tops a leaderboard and finds it weak on their domain. Debug: score the candidate on a labelled slice of real requests before committing.
- Ignoring tokeniser differences. The same prompt costs more or truncates earlier on the chosen model. Debug: tokenise a representative sample with each candidate tokeniser and compare counts.
- Landscape churn as strategy. The team migrates every quarter and never accumulates evaluation history. Debug: count model migrations in the last year and check whether each one improved the frozen benchmark.
In production
Keep a table of approved models with context limits, tokeniser, price unit and evaluation score, and review it on a fixed cadence. Alert when a provider announces a deprecation for anything in the table.
Common misconceptions
People think: Newer always means better for your task.
Actually: Releases optimise different objectives; a newer model can regress on a narrow task or a specific output format.
People think: Parameter count is the main axis.
Actually: Training data, post-training, context handling and serving stack often matter more than size for a given application.
Interview angle
A strong answer organises the space by capability, deployment and constraints, then says how candidates are tested on real traffic. It mentions tokeniser and context differences as integration costs. A weak answer lists model names and recency without saying what task each was chosen for.
Sources
- Liang et al., Holistic Evaluation of Language Models — multi-axis evaluation instead of a single ranking.
- Zhao et al., A Survey of Large Language Models — families, architectures and adaptation methods.
- Chiang et al., Chatbot Arena — preference-based comparison and its limits.