Why it matters
Token pricing looks simple until you try to forecast it. A per-request estimate built from average token counts misses the heavy tail, the retries, the judge calls in your evaluation pipeline, the embedding job that re-runs on every document edit, and the fixed cost of the vector index that sits idle at 3 a.m. Each omission is small; together they routinely explain a gap between forecast and invoice that nobody can attribute.
The second problem is the unit. A request is not a unit of value. If 20% of conversations end with a human taking over, the honest denominator is resolved conversations, and the model call is only part of that cost. Teams that optimise cost per request sometimes make the product worse in ways that raise cost per outcome — shorter context causes more escalations, which cost far more than the tokens saved.
Mental model
A delivery business. Fuel per kilometre is easy to compute and almost irrelevant on its own; what matters is cost per delivered parcel, which includes failed deliveries, redelivery, depot rent and driver time. Fuel is the one line you can measure precisely, which is exactly why it dominates the discussion and not the economics.
The analogy breaks because token spend is metered continuously and visible in near real time, while the outcome cost (a support agent's time, a lost customer) is batched, delayed and estimated. The measurable part is the least important part, which is the trap.
How it works
The interactive figure loads with JavaScript. The text version below describes the same frame.
Figure. Cost per request against monthly volume for four cache-hit rates drawn together, with the fixed-infrastructure share drawn separately against a stated target. The selected rate is filled and the other three stay visible for comparison. Token rates and the request shape are illustrative; the downward slope and the size of the cache offset are the claim.
What to notice. At 60% cache hits a request costs about $0.0107 at two million requests a month and $0.0103 at ten million: fivefold volume moves the number by 3%. The same request without cache hits costs $0.0137, and the $0.015 target then holds only above about half a million requests a month. The cache is the larger lever, and it is bounded by a prefix that must stay stable.
Text version
Cost per request from 0.25 to 10 million monthly requests. Four cost curves are drawn together: no cache falls from $0.0168 at the left edge to $0.0133 at the right, 30% cached from $0.0153 to $0.0118, 60% cached from $0.0138 to $0.0103, and 90% cached from $0.0123 to $0.0088. The 60% curve is filled and thicker because the slider selects it; the other three are thin lines. A dashed line shows fixed infrastructure per request falling from $0.0036 to $0.0001. A horizontal target line sits at $0.015 and a vertical line marks today's two million requests. A cache-hit slider changes which curve is emphasised and reports the cost at today's volume, the saving against no cache and the fixed share.
- Instrument usage per call. Record input tokens, cached input tokens, output tokens (including reasoning tokens where the API bills them), model id, and a purpose tag such as
prod,eval_judge,backfillorretry. - Attach the published rates. Rates are configuration, not constants in code. Store them per provider, per model, per token class, with an effective date, and keep the old rows. See token economics.
- Compute cost per request. Multiply each token class by its rate, then add retrieval, embedding, reranking and any per-call infrastructure fee.
- Add behaviour. Multiply by the retry rate and by the fraction of traffic that is internal (evals, judges, backfills). Those are real spend that never reaches a user.
- Multiply by volume, then by time. Daily requests times 30 is the monthly token volume. Do this separately per route; a global average hides that one feature is 80% of spend.
- Divide by outcomes. Cost per resolved task equals cost per conversation divided by the fraction resolved without escalation, plus the cost of the escalations themselves.
- Add fixed costs. Serving replicas, index storage, observability retention, and idle capacity. Divide by volume to see whether they are noise or the main line.
- Reconcile monthly. Compare the model's output with the provider invoice, per model, and investigate any gap above a tolerance you set. A 5% gap is usually a missing token class; a 30% gap is usually a missing workload.
- Primary unit
- Cost per request
- Decision unit
- Cost per resolved task
- Most volatile input
- Output token count
- Most forgotten input
- Eval and judge calls
- Hardest to attribute
- Shared infrastructure
- Best control
- Per-tenant budget
The arithmetic
Let be fresh input tokens, cached input tokens and output tokens per request, with rates , and expressed per million tokens:
At 200,000 requests per day with 2,400 input tokens and 350 output tokens, monthly volume is:
- input: 200,000 × 2,400 × 30 = 14.4 billion tokens
- output: 200,000 × 350 × 30 = 2.1 billion tokens
So with rates in dollars per million tokens, the monthly variable cost is — 14.4 billion tokens is 14,400 million. That form is more useful than a single number: it shows immediately that a 10% change in output volume moves the bill less than a 10% change in input volume at this ratio, while output remains the more expensive class per token.
Now add behaviour. A 3% retry rate raises variable cost by 3%. If 1,800 of those input tokens are a stable prefix and 60% of requests hit the cache, then 0.6 × 1,800 × 200,000 × 30 = 6.48 billion tokens per month move from to , and the saving is .
Finally divide by outcomes. If 12% of conversations escalate to a human, cost per resolved conversation is plus the escalation cost weighted by 0.12. In most support products the escalation term dominates, which is why a quality improvement that cuts escalations by two points usually beats a 20% token reduction.
A cost function you can test
from dataclasses import dataclass
@dataclass(frozen=True)
class Rates:
input_per_mtok: float
cached_input_per_mtok: float
output_per_mtok: float
def request_cost(usage: dict, r: Rates, retry_rate: float = 0.0) -> float:
fresh = usage["input_tokens"] - usage["cached_input_tokens"]
base = (
fresh / 1e6 * r.input_per_mtok
+ usage["cached_input_tokens"] / 1e6 * r.cached_input_per_mtok
+ usage["output_tokens"] / 1e6 * r.output_per_mtok
)
return base * (1 + retry_rate)
def cost_per_resolved(total_cost: float, conversations: int, escalation_rate: float,
human_cost: float) -> float:
resolved = conversations * (1 - escalation_rate)
return (total_cost + conversations * escalation_rate * human_cost) / resolvedKeep this pure and unit-test it against a handful of recorded usage objects. When the provider changes a rate or adds a token class, the test shows which numbers move.
Trade-offs
| Lever | Saves | Costs | Fails when |
|---|---|---|---|
| Shorter context, less retrieval | Input tokens, latency | Answer quality, more escalations | The dropped context held the answer |
| Prompt caching | Input tokens on repeated prefixes | Cache writes, prefix rigidity | Prompts are personalised per request |
| Smaller model or router | Output and input tokens | Quality, extra routing logic | Hard queries silently degrade |
| Cap output length | Output tokens, latency | Truncated answers | The task needs long structured output |
| Batch API for offline work | Per-token rate | Hours of latency | Anything user-facing |
| Self-hosting | Marginal token cost at high utilisation | GPUs, ops, idle capacity | Utilisation is low or spiky |
Good for
Input tokens, latency
Cost
Answer quality, more escalations
Fails when
The dropped context held the answer
- The dropped context held the answer
- Prompts are personalised per request
- Hard queries silently degrade
- The task needs long structured output
- Anything user-facing
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Shorter context, less retrieval is good for Input tokens, latency, costs Answer quality, more escalations, and fails when The dropped context held the answer. Prompt caching is good for Input tokens on repeated prefixes, costs Cache writes, prefix rigidity, and fails when Prompts are personalised per request. Smaller model or router is good for Output and input tokens, costs Quality, extra routing logic, and fails when Hard queries silently degrade. Cap output length is good for Output tokens, latency, costs Truncated answers, and fails when The task needs long structured output. Batch API for offline work is good for Per-token rate, costs Hours of latency, and fails when Anything user-facing
Failure modes
- Average-token forecasting. Means hide a long tail where a few requests carry enormous context. Debug by bucketing requests by input-token count and computing the share of spend in the top 1%; if it is large, model the tail separately.
- Invisible internal traffic. Judges, backfills, embedding jobs and retries appear in the invoice but not in product dashboards. Debug by grouping spend on the purpose tag and checking that the totals add up to the invoice.
- Cache assumptions that never hit. A prefix that changes per user produces cache writes and no reads. Debug by comparing reported cached tokens against cache-hit metrics; a write-heavy, read-light pattern is the signature.
- Stale rate table. A provider changes a price or introduces a new token class, and the model keeps using last quarter's constants. Debug with a monthly reconciliation per model and a fixed tolerance, escalating any unexplained gap.
In production
Put a budget guardrail in front of the model, not behind it: per-tenant and per-feature caps that refuse or downgrade before the spend happens. See cost budgets. Track unit cost weekly, not monthly; by the time the invoice arrives, the change is six weeks old.
Attribute spend to a tenant and a feature even if billing does not need it. Attribution is how you discover that one integration with an unusual prompt is a third of the bill. Keep a what-if sheet for the three levers you would pull in an incident: cap output, drop to a cheaper model, disable reranking.
Common misconceptions
People think: Input and output tokens cost about the same, so only total volume matters.
Actually: They are priced differently, and output is generated sequentially while input is processed in parallel. Per-token output rates are typically higher, which makes output length both a cost and a latency lever.
People think: Cost per request is the metric to optimise.
Actually: A request that fails and escalates can cost many times a successful one. Cost per resolved task is the number that survives contact with finance and with product.
Interview angle
Expect "how would you estimate the monthly bill for this feature?" A strong answer builds the formula out loud: token classes, volume, retries, internal traffic, retrieval, fixed costs, then divides by outcomes. Then it names the sensitivity: which input moves the number most, and what you would measure first. Weak answers quote a per-request figure without saying what is inside it. See capacity planning and the production interview category.
Sources
- OpenAI API pricing — published per-token rates and cached-input classes; the numbers change, the structure does not.
- Anthropic pricing — a second rate card with the same categories, useful for checking assumptions.
- Pope et al., Efficiently Scaling Transformer Inference — why decode cost scales with output tokens while prefill scales with input.
- FinOps Foundation, FinOps Framework — allocation, forecasting and reconciliation vocabulary that predates LLMs.
Related
- Token economics — what a token is and how counts are produced.
- Cost budgets — enforcement rather than estimation.
- Prompt caching — the biggest single lever on input cost.
- Capacity planning — the same arithmetic for infrastructure.
- Response caching — removing work instead of making it cheaper.