Why it matters
Token spend is driven by three multipliers that are easy to lose track of: prompt length, output length and the number of model calls per user action. An agent that retries a tool five times, or a retrieved context that grows with the corpus, changes cost without changing traffic. A budget makes that visible per unit and gives the system a defined behaviour at the limit, instead of a surprise invoice and an emergency prompt edit.
Mental model
It is a prepaid meter on a rental: you see consumption per unit and the supply stops at zero unless you top up. The analogy breaks because stopping the meter mid-generation produces a broken answer, so the useful design degrades quality before it cuts service.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Cumulative share of spend against share of user actions, cheapest first, with an even-spend diagonal. The two steep segments are the long-context calls and the retry loops; the concentration in the last two per cent is the claim. Costs are illustrative.
What to notice. The median action costs $0.035 and the 99th percentile costs $0.95. The cheapest 78% of actions carry 30% of spend; the next 20 points carry 49%; the dearest 2% carry 21% on their own. A cap aimed at the median changes almost nothing, because the bill is set by the tail that a retry loop or a long context produces.
Text version
Cumulative share of spend against the share of user actions, cheapest first, from 0 to 100 with an even-spend diagonal. The curve rises slowly through the cheap actions, reaching 19% of spend at half of all actions and 30% at 78%. It then climbs steeply through the long-context block, reaching 79% at 98% of actions, and the last 2% of actions take it to 100%. Markers label the 50% and 78% points and a line marks the top 2% with the 21% share of spend. Three steps introduce the median, the middle and the tail; a panel reports the p50 and p99 cost and the top 2% share of spend.
- Define the unit of account, such as cost per resolved ticket or per active tenant, and make sure usage records can be attributed to it.
- Instrument input and output tokens per call, plus calls per user action, and store the model revision with each record.
- Set a budget per unit per period and alerts at fractions of it, routed to the team that can change the behaviour.
- Cap the variables that can run away: maximum output tokens, maximum retrieved context, maximum tool iterations.
- Define a degradation ladder, from a cheaper model to a shorter context to a queued response, and trigger it before the hard stop.
- Review the top spenders monthly against quality, because a cost cut that removes the good answers is not a saving.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Hard cap per tenant | Prevents one customer from exhausting budget | Legitimate spikes get rejected | Usage is genuinely bursty |
| Soft budget with alerts | No user-facing failure | Relies on someone acting | Nobody owns the alert |
| Cheaper model for low-value traffic | Large savings on routine requests | Quality split across the product | Routing cannot classify difficulty |
| Aggressive context truncation | Predictable prompt cost | Answers lose supporting evidence | Tasks need long documents |
Good for
Prevents one customer from exhausting budget
Cost
Legitimate spikes get rejected
Fails when
Usage is genuinely bursty
- Usage is genuinely bursty
- Nobody owns the alert
- Routing cannot classify difficulty
- Tasks need long documents
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Hard cap per tenant is good for Prevents one customer from exhausting budget, costs Legitimate spikes get rejected, and fails when Usage is genuinely bursty. Soft budget with alerts is good for No user-facing failure, costs Relies on someone acting, and fails when Nobody owns the alert. Cheaper model for low-value traffic is good for Large savings on routine requests, costs Quality split across the product, and fails when Routing cannot classify difficulty. Aggressive context truncation is good for Predictable prompt cost, costs Answers lose supporting evidence, and fails when Tasks need long documents
Failure modes
- Unattributed spend. The bill rises and no dashboard can say which feature caused it. Debug: sample requests and check that each usage record carries tenant, feature and prompt version.
- Retry loop burn. A failing tool call is retried until the step cap, multiplying cost per request. Debug: histogram model calls per user action and inspect the tail.
- Cap without a message. The limit is reached and the user sees a generic error with no path forward. Debug: exercise the cap in staging and read the response the client receives.
In production
Publish cost per unit next to latency and error rate, and alert on the derivative as well as the total. Rehearse the degradation ladder, because a budget that has never been hit is a hypothesis.
Common misconceptions
People think: Cost is a finance concern, not an engineering one.
Actually: Every cost driver is a design choice in the request path: context size, output cap, retries and model routing.
People think: A cheaper model always reduces cost per task.
Actually: If it needs more retries or longer prompts to succeed, cost per resolved task can rise while cost per token falls.
Interview angle
Define the unit of account first, then the three multipliers and the cap that bounds each. A strong answer includes a degradation ladder and an attribution scheme. Weak answers talk about reducing spend without naming what one request costs.
Sources
- FinOps Foundation, FinOps Framework — unit economics, allocation and budget policy.
- Google Cloud, Cost optimization — attribution and right-sizing practice.
- OpenTelemetry GenAI semantic conventions — token usage attributes that make cost attributable.