Why it matters
Model spend is easy to measure and hard to attribute. A single request hides prompt construction, retries, tool loops and the difference between cached and fresh input. Teams discover the bill after launch, when the expensive path is already the default. Treating tokens as the unit of cost, and modelling the request as a sum of priced components, makes the expensive path visible before it ships.
Mental model
Token economics is a taxi meter with two rates: distance and waiting time. The comparison breaks because the meter also charges for the route you describe at the start, and describing it twice costs as much as driving part of it.
How it works
- 1Instrument every call with input, cached input, output and reasoning token counts, plus the model id.
- 2Compute cost per request and per feature from those counts, using current provider rates in one configuration file.
- 3Attribute cost to a route or feature, so a dashboard shows which product surface spends the budget.
- 4Model the marginal cost of a change, such as a longer prompt, an extra retry or an extra tool step, before shipping it.
- 5Reduce in order of leverage: shorter prompts, caching stable prefixes, fewer retries, smaller models on easy routes.
- 6Set per-request and per-tenant caps, and alert when spend per successful request drifts.
Step 1 of 6
Instrument every call with input, cached input, output and reasoning token counts, plus the model id.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Instrument every call with input, cached input, output and reasoning token counts, plus t… 2. Compute cost per request and per feature from those counts, using current provider rates … 3. Attribute cost to a route or feature, so a dashboard shows which product surface spends t… 4. Model the marginal cost of a change, such as a longer prompt, an extra retry or an extra … 5. Reduce in order of leverage: shorter prompts, caching stable prefixes, fewer retries, sma… 6. Set per-request and per-tenant caps, and alert when spend per successful request drifts.
- Instrument every call with input, cached input, output and reasoning token counts, plus the model id.
- Compute cost per request and per feature from those counts, using current provider rates in one configuration file.
- Attribute cost to a route or feature, so a dashboard shows which product surface spends the budget.
- Model the marginal cost of a change, such as a longer prompt, an extra retry or an extra tool step, before shipping it.
- Reduce in order of leverage: shorter prompts, caching stable prefixes, fewer retries, smaller models on easy routes.
- Set per-request and per-tenant caps, and alert when spend per successful request drifts.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Long prompt with full context | Better answers on complex tasks | Input cost and prefill latency | The same prefix is resent without caching |
| Aggressive truncation | Cheap and fast | Missing evidence and worse answers | The task needs long-range context |
| Small model for most traffic | Low cost per request | Quality drop on hard cases | You cannot route by difficulty |
| Cascade to a large model | Cost control with a quality floor | Two calls for escalated requests | Escalation is triggered too often |
Good for
Better answers on complex tasks
Cost
Input cost and prefill latency
Fails when
The same prefix is resent without caching
- The same prefix is resent without caching
- The task needs long-range context
- You cannot route by difficulty
- Escalation is triggered too often
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Long prompt with full context is good for Better answers on complex tasks, costs Input cost and prefill latency, and fails when The same prefix is resent without caching. Aggressive truncation is good for Cheap and fast, costs Missing evidence and worse answers, and fails when The task needs long-range context. Small model for most traffic is good for Low cost per request, costs Quality drop on hard cases, and fails when You cannot route by difficulty. Cascade to a large model is good for Cost control with a quality floor, costs Two calls for escalated requests, and fails when Escalation is triggered too often
Failure modes
- Unattributed spend. The bill rises and no one can say which feature caused it. Debug: group cost by route and model for one day and compare with the previous week.
- Retry multiplier. A transient error doubles cost because each attempt is billed. Debug: count provider calls per user request and compare with successful responses.
- Cache-blind accounting. Cached input is priced as fresh input and the saving never appears. Debug: reconcile usage fields against the invoice and check cached token counts are recorded.
In production
Alert on cost per successful request and on the share of spend from retries and tool loops. Recalculate unit economics when prices or model routing change, not only at planning time.
Common misconceptions
People think: Cost is proportional to the number of users.
Actually: It is proportional to tokens per request and requests per user, so a prompt change can move it more than growth.
People think: Output length is a minor cost.
Actually: Output tokens are generated serially, priced higher in most APIs, and drive latency, so they are usually the dominant term.
Interview angle
Build the cost model from tokens per component, then show how caching, retries and routing change it. Strong answers attribute spend to features and name the dominant term. Weak answers quote a price per million tokens without saying how many tokens a request actually uses.
Sources
- Anthropic, Pricing — input, output and cache pricing units.
- Pope et al., Efficiently Scaling Transformer Inference — why prefill and decode have different costs.
- OpenAI, Pricing — per-token billing across model tiers.