Why it matters
A team that compares only per-token prices will self-host a model, discover that peak capacity sits idle most of the day, and pay for that idle time in reserved device hours. The reverse mistake is just as common: staying hosted through a residency requirement or a version freeze the provider cannot satisfy. The decision is a capacity-planning problem wearing the costume of a purchasing decision.
Mental model
Self-hosting is buying a delivery van instead of taking taxis. It wins once you drive the same route every day and lose the fare meter; it loses when the van sits in the driveway and you still pay insurance. The comparison breaks because a taxi fleet can also refuse your package, which is the data-terms argument the van cannot carry.
How it works
- 1Measure monthly input and output tokens plus peak concurrent requests, not average requests per second.
- 2Convert that load into device-hours using throughput measured on your own prompt and output lengths.
- 3Price the same load both ways, including idle capacity, egress, storage and the on-call time self-hosting adds.
- 4List the requirements a provider cannot meet: residency, weight access, a pinned checkpoint, a custom kernel or fine-tuned adapters.
- 5Pilot the smaller workload on the candidate path and compare tail latency and quality on the same evaluation set.
- 6Decide the fallback direction in advance, so a capacity shortfall or a provider deprecation has a rehearsed answer.
Step 1 of 6
Measure monthly input and output tokens plus peak concurrent requests, not average requests per second.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Measure monthly input and output tokens plus peak concurrent requests, not average reques… 2. Convert that load into device-hours using throughput measured on your own prompt and outp… 3. Price the same load both ways, including idle capacity, egress, storage and the on-call t… 4. List the requirements a provider cannot meet: residency, weight access, a pinned checkpoi… 5. Pilot the smaller workload on the candidate path and compare tail latency and quality on … 6. Decide the fallback direction in advance, so a capacity shortfall or a provider deprecati…
- Measure monthly input and output tokens plus peak concurrent requests, not average requests per second.
- Convert that load into device-hours using throughput measured on your own prompt and output lengths.
- Price the same load both ways, including idle capacity, egress, storage and the on-call time self-hosting adds.
- List the requirements a provider cannot meet: residency, weight access, a pinned checkpoint, a custom kernel or fine-tuned adapters.
- Pilot the smaller workload on the candidate path and compare tail latency and quality on the same evaluation set.
- Decide the fallback direction in advance, so a capacity shortfall or a provider deprecation has a rehearsed answer.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Hosted API | No capacity planning, instant access to new models | Per-token spend and data leaving your boundary | Volume is steady and high enough to own capacity |
| Self-hosted open weights | Fixed cost, residency, version pinning | Utilisation risk and device operations | Load is spiky or the team has no inference on-call |
| Reserved capacity | Lower unit cost for predictable load | Paid whether used or not | The demand forecast is uncertain |
| Hybrid with hosted burst | Handles peaks without over-provisioning | Two stacks and two evaluation paths | The self-hosted baseline is already unstable |
Good for
No capacity planning, instant access to new models
Cost
Per-token spend and data leaving your boundary
Fails when
Volume is steady and high enough to own capacity
- Volume is steady and high enough to own capacity
- Load is spiky or the team has no inference on-call
- The demand forecast is uncertain
- The self-hosted baseline is already unstable
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Hosted API is good for No capacity planning, instant access to new models, costs Per-token spend and data leaving your boundary, and fails when Volume is steady and high enough to own capacity. Self-hosted open weights is good for Fixed cost, residency, version pinning, costs Utilisation risk and device operations, and fails when Load is spiky or the team has no inference on-call. Reserved capacity is good for Lower unit cost for predictable load, costs Paid whether used or not, and fails when The demand forecast is uncertain. Hybrid with hosted burst is good for Handles peaks without over-provisioning, costs Two stacks and two evaluation paths, and fails when The self-hosted baseline is already unstable
Failure modes
- Utilisation illusion. The cluster looks cheap per token because the calculation divides by capacity rather than by tokens actually served. Debug: divide monthly device spend by billed tokens and compare with the hosted invoice.
- Version freeze by accident. A self-hosted deployment stays on an old checkpoint because upgrading means re-validating kernels and quantisation. Debug: put the model revision in the release calendar and treat an upgrade as a project.
- Standby that cannot absorb load. Failover to the hosted path hits a quota nobody raised for full traffic. Debug: load-test the failover provider at production concurrency.
In production
Track cost per thousand tokens, device duty cycle and queue wait together, because a cheap idle cluster and an overloaded one look identical on a spend chart. Re-test the hosted fallback at production concurrency each quarter.
Common misconceptions
People think: Self-hosting is cheaper at any scale.
Actually: It is cheaper above the utilisation where fixed device cost divided by served tokens beats the per-token price, and that point depends on your duty cycle.
People think: Hosted providers cannot meet compliance requirements.
Actually: Many offer residency, retention controls and private networking; the question is whether the specific terms match your obligation.
Interview angle
Frame the decision as utilisation, control and risk: tokens per month, peak concurrency, requirements a provider cannot contract to. Show the arithmetic that makes the crossover visible, and name the operational burden you would be buying. Weak answers compare a price list without mentioning duty cycle or who gets paged at night.
Sources
- vLLM documentation — measuring throughput on your own workload before sizing hardware.
- FinOps Foundation, FinOps Framework — unit economics and cost allocation for infrastructure spend.
- Google Cloud, Cost optimization — right-sizing and commitment decisions for compute.