Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 4 of 28

Open vs closed

0 of 28 complete

Why it matters

The choice between open-weight and closed models is usually framed as quality versus cost, but the binding constraints are different: whether data can leave your network, whether you can staff inference operations, whether the licence permits your use, and what happens when a provider retires the model. Those questions decide whether a technically superior closed model is usable at all in a regulated or latency-sensitive product.

Mental model

Closed models are utilities; open weights are a generator you install. The utility bills per unit and handles maintenance; the generator needs a room, fuel and an engineer. The comparison breaks because the utility can change its output mix without notice, while your generator keeps running the version you installed.

How it works

Mechanism, step by stepInteractive
  1. 1Write the constraints first: data residency, retention, licence compatibility, latency ceiling and the team's on-call capacity.
  2. 2Check the licence on the weights, including acceptable-use and redistribution terms, before benchmarking.
  3. 3Estimate total cost for open weights from GPU hours at realistic utilisation plus engineering time for serving, upgrades and evaluation.
  4. 4Test both options on the same evaluation set with the same prompts, and record quality per slice rather than an average.
  5. 5Decide the fallback direction, since a closed primary with an open fallback and the reverse imply different interfaces.
  6. 6Revisit when volume, regulation or the model's context limit changes materially.

Step 1 of 6

Write the constraints first: data residency, retention, licence compatibility, latency ceiling and the team's on-call capacity.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Write the constraints first: data residency, retention, licence compatibility, latency ce… 2. Check the licence on the weights, including acceptable-use and redistribution terms, befo… 3. Estimate total cost for open weights from GPU hours at realistic utilisation plus enginee… 4. Test both options on the same evaluation set with the same prompts, and record quality pe… 5. Decide the fallback direction, since a closed primary with an open fallback and the rever… 6. Revisit when volume, regulation or the model's context limit changes materially.

  1. Write the constraints first: data residency, retention, licence compatibility, latency ceiling and the team's on-call capacity.
  2. Check the licence on the weights, including acceptable-use and redistribution terms, before benchmarking.
  3. Estimate total cost for open weights from GPU hours at realistic utilisation plus engineering time for serving, upgrades and evaluation.
  4. Test both options on the same evaluation set with the same prompts, and record quality per slice rather than an average.
  5. Decide the fallback direction, since a closed primary with an open fallback and the reverse imply different interfaces.
  6. Revisit when volume, regulation or the model's context limit changes materially.

Trade-offs

ChoiceGood forCostFails when
Closed APIBest capability per engineering hourPer-token price, no weight control, deprecation riskData cannot leave your boundary
Self-hosted open weightsData stays inside, latency predictableGPU capital, utilisation risk, upgrade workTraffic is spiky or the team is small
Managed open-weight endpointNo GPU operations, weight choiceProvider still in the path, less tuningYou need custom kernels or quantisation
Fine-tuned open modelTask-specific quality, small footprintTraining and evaluation pipelineThe task shifts faster than retraining cycles
Trade-off mapInteractive

Good for

Best capability per engineering hour

Cost

Per-token price, no weight control, deprecation risk

Fails when

Data cannot leave your boundary

  1. Data cannot leave your boundary
  2. Traffic is spiky or the team is small
  3. You need custom kernels or quantisation
  4. The task shifts faster than retraining cycles

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Closed API is good for Best capability per engineering hour, costs Per-token price, no weight control, deprecation risk, and fails when Data cannot leave your boundary. Self-hosted open weights is good for Data stays inside, latency predictable, costs GPU capital, utilisation risk, upgrade work, and fails when Traffic is spiky or the team is small. Managed open-weight endpoint is good for No GPU operations, weight choice, costs Provider still in the path, less tuning, and fails when You need custom kernels or quantisation. Fine-tuned open model is good for Task-specific quality, small footprint, costs Training and evaluation pipeline, and fails when The task shifts faster than retraining cycles

Failure modes

  • Licence discovered late. A model that benchmarks well cannot be used commercially or must be redistributed. Debug: read the licence before the bake-off and record the terms next to the evaluation.
  • Utilisation blindness. Self-hosting looks cheaper per token but GPUs sit idle most of the day. Debug: compute cost at your actual peak-to-average ratio, not at saturation.
  • Deprecation exposure. A closed model is retired and the prompt does not transfer. Debug: keep a tested open fallback and score it on the same set every quarter.

In production

Monitor cost per successful request for both paths and keep the fallback warm enough to be real. For self-hosted models, track queue depth and time to first token as capacity signals, not just error rate.

Common misconceptions

People think: Open weights mean open training data and reproducible training.

Actually: Most releases publish weights and a model card, not the data mixture, compute or full recipe.

People think: Self-hosting is always cheaper at scale.

Actually: Only above a utilisation threshold; below it, idle GPUs cost more than per-token pricing.

Interview angle

Strong answers start from constraints, not quality: data path, licence, staffing and deprecation. They price self-hosting at realistic utilisation and describe a tested fallback. Weak answers compare a benchmark score and assume the cheaper per-token number is the cheaper system.

Sources