Skip to content
AI Engineer Guide.

concept · operations · intermediate · concept 17 of 38

Cost modelling

0 of 38 complete

Why it matters

Token pricing looks simple until you try to forecast it. A per-request estimate built from average token counts misses the heavy tail, the retries, the judge calls in your evaluation pipeline, the embedding job that re-runs on every document edit, and the fixed cost of the vector index that sits idle at 3 a.m. Each omission is small; together they routinely explain a gap between forecast and invoice that nobody can attribute.

The second problem is the unit. A request is not a unit of value. If 20% of conversations end with a human taking over, the honest denominator is resolved conversations, and the model call is only part of that cost. Teams that optimise cost per request sometimes make the product worse in ways that raise cost per outcome — shorter context causes more escalations, which cost far more than the tokens saved.

Mental model

A delivery business. Fuel per kilometre is easy to compute and almost irrelevant on its own; what matters is cost per delivered parcel, which includes failed deliveries, redelivery, depot rent and driver time. Fuel is the one line you can measure precisely, which is exactly why it dominates the discussion and not the economics.

The analogy breaks because token spend is metered continuously and visible in near real time, while the outcome cost (a support agent's time, a lost customer) is batched, delayed and estimated. The measurable part is the least important part, which is the trap.

How it works

Cost per request is a curve, not a numberInteractive

The interactive figure loads with JavaScript. The text version below describes the same frame.

Figure. Cost per request against monthly volume for four cache-hit rates drawn together, with the fixed-infrastructure share drawn separately against a stated target. The selected rate is filled and the other three stay visible for comparison. Token rates and the request shape are illustrative; the downward slope and the size of the cache offset are the claim.

What to notice. At 60% cache hits a request costs about $0.0107 at two million requests a month and $0.0103 at ten million: fivefold volume moves the number by 3%. The same request without cache hits costs $0.0137, and the $0.015 target then holds only above about half a million requests a month. The cache is the larger lever, and it is bounded by a prefix that must stay stable.

Text version

Cost per request from 0.25 to 10 million monthly requests. Four cost curves are drawn together: no cache falls from $0.0168 at the left edge to $0.0133 at the right, 30% cached from $0.0153 to $0.0118, 60% cached from $0.0138 to $0.0103, and 90% cached from $0.0123 to $0.0088. The 60% curve is filled and thicker because the slider selects it; the other three are thin lines. A dashed line shows fixed infrastructure per request falling from $0.0036 to $0.0001. A horizontal target line sits at $0.015 and a vertical line marks today's two million requests. A cache-hit slider changes which curve is emphasised and reports the cost at today's volume, the saving against no cache and the fixed share.

  1. Instrument usage per call. Record input tokens, cached input tokens, output tokens (including reasoning tokens where the API bills them), model id, and a purpose tag such as prod, eval_judge, backfill or retry.
  2. Attach the published rates. Rates are configuration, not constants in code. Store them per provider, per model, per token class, with an effective date, and keep the old rows. See token economics.
  3. Compute cost per request. Multiply each token class by its rate, then add retrieval, embedding, reranking and any per-call infrastructure fee.
  4. Add behaviour. Multiply by the retry rate and by the fraction of traffic that is internal (evals, judges, backfills). Those are real spend that never reaches a user.
  5. Multiply by volume, then by time. Daily requests times 30 is the monthly token volume. Do this separately per route; a global average hides that one feature is 80% of spend.
  6. Divide by outcomes. Cost per resolved task equals cost per conversation divided by the fraction resolved without escalation, plus the cost of the escalations themselves.
  7. Add fixed costs. Serving replicas, index storage, observability retention, and idle capacity. Divide by volume to see whether they are noise or the main line.
  8. Reconcile monthly. Compare the model's output with the provider invoice, per model, and investigate any gap above a tolerance you set. A 5% gap is usually a missing token class; a 30% gap is usually a missing workload.
Primary unit
Cost per request
Decision unit
Cost per resolved task
Most volatile input
Output token count
Most forgotten input
Eval and judge calls
Hardest to attribute
Shared infrastructure
Best control
Per-tenant budget

The arithmetic

Let TinT_{in} be fresh input tokens, TcacheT_{cache} cached input tokens and ToutT_{out} output tokens per request, with rates pinp_{in}, pcachep_{cache} and poutp_{out} expressed per million tokens:

Creq=Tin⋅pin+Tcache⋅pcache+Tout⋅pout106+CretrievalC_{\text{req}} = \frac{T_{in} \cdot p_{in} + T_{cache} \cdot p_{cache} + T_{out} \cdot p_{out}}{10^6} + C_{\text{retrieval}}

At 200,000 requests per day with 2,400 input tokens and 350 output tokens, monthly volume is:

  • input: 200,000 × 2,400 × 30 = 14.4 billion tokens
  • output: 200,000 × 350 × 30 = 2.1 billion tokens

So with rates in dollars per million tokens, the monthly variable cost is 14,400⋅pin+2,100⋅pout14{,}400 \cdot p_{in} + 2{,}100 \cdot p_{out} — 14.4 billion tokens is 14,400 million. That form is more useful than a single number: it shows immediately that a 10% change in output volume moves the bill less than a 10% change in input volume at this ratio, while output remains the more expensive class per token.

Now add behaviour. A 3% retry rate raises variable cost by 3%. If 1,800 of those input tokens are a stable prefix and 60% of requests hit the cache, then 0.6 × 1,800 × 200,000 × 30 = 6.48 billion tokens per month move from pinp_{in} to pcachep_{cache}, and the saving is 6,480⋅(pin−pcache)6{,}480 \cdot (p_{in} - p_{cache}).

Finally divide by outcomes. If 12% of conversations escalate to a human, cost per resolved conversation is Cconv/0.88C_{\text{conv}} / 0.88 plus the escalation cost weighted by 0.12. In most support products the escalation term dominates, which is why a quality improvement that cuts escalations by two points usually beats a 20% token reduction.

A cost function you can test

from dataclasses import dataclass
 
 
@dataclass(frozen=True)
class Rates:
    input_per_mtok: float
    cached_input_per_mtok: float
    output_per_mtok: float
 
 
def request_cost(usage: dict, r: Rates, retry_rate: float = 0.0) -> float:
    fresh = usage["input_tokens"] - usage["cached_input_tokens"]
    base = (
        fresh / 1e6 * r.input_per_mtok
        + usage["cached_input_tokens"] / 1e6 * r.cached_input_per_mtok
        + usage["output_tokens"] / 1e6 * r.output_per_mtok
    )
    return base * (1 + retry_rate)
 
 
def cost_per_resolved(total_cost: float, conversations: int, escalation_rate: float,
                      human_cost: float) -> float:
    resolved = conversations * (1 - escalation_rate)
    return (total_cost + conversations * escalation_rate * human_cost) / resolved

Keep this pure and unit-test it against a handful of recorded usage objects. When the provider changes a rate or adds a token class, the test shows which numbers move.

Trade-offs

LeverSavesCostsFails when
Shorter context, less retrievalInput tokens, latencyAnswer quality, more escalationsThe dropped context held the answer
Prompt cachingInput tokens on repeated prefixesCache writes, prefix rigidityPrompts are personalised per request
Smaller model or routerOutput and input tokensQuality, extra routing logicHard queries silently degrade
Cap output lengthOutput tokens, latencyTruncated answersThe task needs long structured output
Batch API for offline workPer-token rateHours of latencyAnything user-facing
Self-hostingMarginal token cost at high utilisationGPUs, ops, idle capacityUtilisation is low or spiky
Trade-off mapInteractive

Good for

Input tokens, latency

Cost

Answer quality, more escalations

Fails when

The dropped context held the answer

  1. The dropped context held the answer
  2. Prompts are personalised per request
  3. Hard queries silently degrade
  4. The task needs long structured output
  5. Anything user-facing

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Shorter context, less retrieval is good for Input tokens, latency, costs Answer quality, more escalations, and fails when The dropped context held the answer. Prompt caching is good for Input tokens on repeated prefixes, costs Cache writes, prefix rigidity, and fails when Prompts are personalised per request. Smaller model or router is good for Output and input tokens, costs Quality, extra routing logic, and fails when Hard queries silently degrade. Cap output length is good for Output tokens, latency, costs Truncated answers, and fails when The task needs long structured output. Batch API for offline work is good for Per-token rate, costs Hours of latency, and fails when Anything user-facing

Failure modes

  • Average-token forecasting. Means hide a long tail where a few requests carry enormous context. Debug by bucketing requests by input-token count and computing the share of spend in the top 1%; if it is large, model the tail separately.
  • Invisible internal traffic. Judges, backfills, embedding jobs and retries appear in the invoice but not in product dashboards. Debug by grouping spend on the purpose tag and checking that the totals add up to the invoice.
  • Cache assumptions that never hit. A prefix that changes per user produces cache writes and no reads. Debug by comparing reported cached tokens against cache-hit metrics; a write-heavy, read-light pattern is the signature.
  • Stale rate table. A provider changes a price or introduces a new token class, and the model keeps using last quarter's constants. Debug with a monthly reconciliation per model and a fixed tolerance, escalating any unexplained gap.

In production

Put a budget guardrail in front of the model, not behind it: per-tenant and per-feature caps that refuse or downgrade before the spend happens. See cost budgets. Track unit cost weekly, not monthly; by the time the invoice arrives, the change is six weeks old.

Attribute spend to a tenant and a feature even if billing does not need it. Attribution is how you discover that one integration with an unusual prompt is a third of the bill. Keep a what-if sheet for the three levers you would pull in an incident: cap output, drop to a cheaper model, disable reranking.

Common misconceptions

People think: Input and output tokens cost about the same, so only total volume matters.

Actually: They are priced differently, and output is generated sequentially while input is processed in parallel. Per-token output rates are typically higher, which makes output length both a cost and a latency lever.

People think: Cost per request is the metric to optimise.

Actually: A request that fails and escalates can cost many times a successful one. Cost per resolved task is the number that survives contact with finance and with product.

Interview angle

Expect "how would you estimate the monthly bill for this feature?" A strong answer builds the formula out loud: token classes, volume, retries, internal traffic, retrieval, fixed costs, then divides by outcomes. Then it names the sensitivity: which input moves the number most, and what you would measure first. Weak answers quote a per-request figure without saying what is inside it. See capacity planning and the production interview category.

Sources

  • Token economics — what a token is and how counts are produced.
  • Cost budgets — enforcement rather than estimation.
  • Prompt caching — the biggest single lever on input cost.
  • Capacity planning — the same arithmetic for infrastructure.
  • Response caching — removing work instead of making it cheaper.