Why it matters
The choice between open-weight and closed models is usually framed as quality versus cost, but the binding constraints are different: whether data can leave your network, whether you can staff inference operations, whether the licence permits your use, and what happens when a provider retires the model. Those questions decide whether a technically superior closed model is usable at all in a regulated or latency-sensitive product.
Mental model
Closed models are utilities; open weights are a generator you install. The utility bills per unit and handles maintenance; the generator needs a room, fuel and an engineer. The comparison breaks because the utility can change its output mix without notice, while your generator keeps running the version you installed.
How it works
- 1Write the constraints first: data residency, retention, licence compatibility, latency ceiling and the team's on-call capacity.
- 2Check the licence on the weights, including acceptable-use and redistribution terms, before benchmarking.
- 3Estimate total cost for open weights from GPU hours at realistic utilisation plus engineering time for serving, upgrades and evaluation.
- 4Test both options on the same evaluation set with the same prompts, and record quality per slice rather than an average.
- 5Decide the fallback direction, since a closed primary with an open fallback and the reverse imply different interfaces.
- 6Revisit when volume, regulation or the model's context limit changes materially.
Step 1 of 6
Write the constraints first: data residency, retention, licence compatibility, latency ceiling and the team's on-call capacity.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Write the constraints first: data residency, retention, licence compatibility, latency ce… 2. Check the licence on the weights, including acceptable-use and redistribution terms, befo… 3. Estimate total cost for open weights from GPU hours at realistic utilisation plus enginee… 4. Test both options on the same evaluation set with the same prompts, and record quality pe… 5. Decide the fallback direction, since a closed primary with an open fallback and the rever… 6. Revisit when volume, regulation or the model's context limit changes materially.
- Write the constraints first: data residency, retention, licence compatibility, latency ceiling and the team's on-call capacity.
- Check the licence on the weights, including acceptable-use and redistribution terms, before benchmarking.
- Estimate total cost for open weights from GPU hours at realistic utilisation plus engineering time for serving, upgrades and evaluation.
- Test both options on the same evaluation set with the same prompts, and record quality per slice rather than an average.
- Decide the fallback direction, since a closed primary with an open fallback and the reverse imply different interfaces.
- Revisit when volume, regulation or the model's context limit changes materially.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Closed API | Best capability per engineering hour | Per-token price, no weight control, deprecation risk | Data cannot leave your boundary |
| Self-hosted open weights | Data stays inside, latency predictable | GPU capital, utilisation risk, upgrade work | Traffic is spiky or the team is small |
| Managed open-weight endpoint | No GPU operations, weight choice | Provider still in the path, less tuning | You need custom kernels or quantisation |
| Fine-tuned open model | Task-specific quality, small footprint | Training and evaluation pipeline | The task shifts faster than retraining cycles |
Good for
Best capability per engineering hour
Cost
Per-token price, no weight control, deprecation risk
Fails when
Data cannot leave your boundary
- Data cannot leave your boundary
- Traffic is spiky or the team is small
- You need custom kernels or quantisation
- The task shifts faster than retraining cycles
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Closed API is good for Best capability per engineering hour, costs Per-token price, no weight control, deprecation risk, and fails when Data cannot leave your boundary. Self-hosted open weights is good for Data stays inside, latency predictable, costs GPU capital, utilisation risk, upgrade work, and fails when Traffic is spiky or the team is small. Managed open-weight endpoint is good for No GPU operations, weight choice, costs Provider still in the path, less tuning, and fails when You need custom kernels or quantisation. Fine-tuned open model is good for Task-specific quality, small footprint, costs Training and evaluation pipeline, and fails when The task shifts faster than retraining cycles
Failure modes
- Licence discovered late. A model that benchmarks well cannot be used commercially or must be redistributed. Debug: read the licence before the bake-off and record the terms next to the evaluation.
- Utilisation blindness. Self-hosting looks cheaper per token but GPUs sit idle most of the day. Debug: compute cost at your actual peak-to-average ratio, not at saturation.
- Deprecation exposure. A closed model is retired and the prompt does not transfer. Debug: keep a tested open fallback and score it on the same set every quarter.
In production
Monitor cost per successful request for both paths and keep the fallback warm enough to be real. For self-hosted models, track queue depth and time to first token as capacity signals, not just error rate.
Common misconceptions
People think: Open weights mean open training data and reproducible training.
Actually: Most releases publish weights and a model card, not the data mixture, compute or full recipe.
People think: Self-hosting is always cheaper at scale.
Actually: Only above a utilisation threshold; below it, idle GPUs cost more than per-token pricing.
Interview angle
Strong answers start from constraints, not quality: data path, licence, staffing and deprecation. They price self-hosting at realistic utilisation and describe a tested fallback. Weak answers compare a benchmark score and assume the cheaper per-token number is the cheaper system.
Sources
- Kapoor et al., On the Societal Impact of Open Foundation Models — what open weights do and do not enable.
- Grattafiori et al., The Llama 3 Herd of Models — a concrete open-weight release with licence and serving details.
- Bommasani et al., The Foundation Model Transparency Index — how much training information releases actually disclose.