Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 3 of 38

Hosted vs self-hosted

0 of 38 complete

Why it matters

A team that compares only per-token prices will self-host a model, discover that peak capacity sits idle most of the day, and pay for that idle time in reserved device hours. The reverse mistake is just as common: staying hosted through a residency requirement or a version freeze the provider cannot satisfy. The decision is a capacity-planning problem wearing the costume of a purchasing decision.

Mental model

Self-hosting is buying a delivery van instead of taking taxis. It wins once you drive the same route every day and lose the fare meter; it loses when the van sits in the driveway and you still pay insurance. The comparison breaks because a taxi fleet can also refuse your package, which is the data-terms argument the van cannot carry.

How it works

Mechanism, step by stepInteractive
  1. 1Measure monthly input and output tokens plus peak concurrent requests, not average requests per second.
  2. 2Convert that load into device-hours using throughput measured on your own prompt and output lengths.
  3. 3Price the same load both ways, including idle capacity, egress, storage and the on-call time self-hosting adds.
  4. 4List the requirements a provider cannot meet: residency, weight access, a pinned checkpoint, a custom kernel or fine-tuned adapters.
  5. 5Pilot the smaller workload on the candidate path and compare tail latency and quality on the same evaluation set.
  6. 6Decide the fallback direction in advance, so a capacity shortfall or a provider deprecation has a rehearsed answer.

Step 1 of 6

Measure monthly input and output tokens plus peak concurrent requests, not average requests per second.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Measure monthly input and output tokens plus peak concurrent requests, not average reques… 2. Convert that load into device-hours using throughput measured on your own prompt and outp… 3. Price the same load both ways, including idle capacity, egress, storage and the on-call t… 4. List the requirements a provider cannot meet: residency, weight access, a pinned checkpoi… 5. Pilot the smaller workload on the candidate path and compare tail latency and quality on … 6. Decide the fallback direction in advance, so a capacity shortfall or a provider deprecati…

  1. Measure monthly input and output tokens plus peak concurrent requests, not average requests per second.
  2. Convert that load into device-hours using throughput measured on your own prompt and output lengths.
  3. Price the same load both ways, including idle capacity, egress, storage and the on-call time self-hosting adds.
  4. List the requirements a provider cannot meet: residency, weight access, a pinned checkpoint, a custom kernel or fine-tuned adapters.
  5. Pilot the smaller workload on the candidate path and compare tail latency and quality on the same evaluation set.
  6. Decide the fallback direction in advance, so a capacity shortfall or a provider deprecation has a rehearsed answer.

Trade-offs

ChoiceGood forCostFails when
Hosted APINo capacity planning, instant access to new modelsPer-token spend and data leaving your boundaryVolume is steady and high enough to own capacity
Self-hosted open weightsFixed cost, residency, version pinningUtilisation risk and device operationsLoad is spiky or the team has no inference on-call
Reserved capacityLower unit cost for predictable loadPaid whether used or notThe demand forecast is uncertain
Hybrid with hosted burstHandles peaks without over-provisioningTwo stacks and two evaluation pathsThe self-hosted baseline is already unstable
Trade-off mapInteractive

Good for

No capacity planning, instant access to new models

Cost

Per-token spend and data leaving your boundary

Fails when

Volume is steady and high enough to own capacity

  1. Volume is steady and high enough to own capacity
  2. Load is spiky or the team has no inference on-call
  3. The demand forecast is uncertain
  4. The self-hosted baseline is already unstable

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Hosted API is good for No capacity planning, instant access to new models, costs Per-token spend and data leaving your boundary, and fails when Volume is steady and high enough to own capacity. Self-hosted open weights is good for Fixed cost, residency, version pinning, costs Utilisation risk and device operations, and fails when Load is spiky or the team has no inference on-call. Reserved capacity is good for Lower unit cost for predictable load, costs Paid whether used or not, and fails when The demand forecast is uncertain. Hybrid with hosted burst is good for Handles peaks without over-provisioning, costs Two stacks and two evaluation paths, and fails when The self-hosted baseline is already unstable

Failure modes

  • Utilisation illusion. The cluster looks cheap per token because the calculation divides by capacity rather than by tokens actually served. Debug: divide monthly device spend by billed tokens and compare with the hosted invoice.
  • Version freeze by accident. A self-hosted deployment stays on an old checkpoint because upgrading means re-validating kernels and quantisation. Debug: put the model revision in the release calendar and treat an upgrade as a project.
  • Standby that cannot absorb load. Failover to the hosted path hits a quota nobody raised for full traffic. Debug: load-test the failover provider at production concurrency.

In production

Track cost per thousand tokens, device duty cycle and queue wait together, because a cheap idle cluster and an overloaded one look identical on a spend chart. Re-test the hosted fallback at production concurrency each quarter.

Common misconceptions

People think: Self-hosting is cheaper at any scale.

Actually: It is cheaper above the utilisation where fixed device cost divided by served tokens beats the per-token price, and that point depends on your duty cycle.

People think: Hosted providers cannot meet compliance requirements.

Actually: Many offer residency, retention controls and private networking; the question is whether the specific terms match your obligation.

Interview angle

Frame the decision as utilisation, control and risk: tokens per month, peak concurrency, requirements a provider cannot contract to. Show the arithmetic that makes the crossover visible, and name the operational burden you would be buying. Weak answers compare a price list without mentioning duty cycle or who gets paged at night.

Sources