Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 28 of 28

Token economics

0 of 28 complete

Why it matters

Model spend is easy to measure and hard to attribute. A single request hides prompt construction, retries, tool loops and the difference between cached and fresh input. Teams discover the bill after launch, when the expensive path is already the default. Treating tokens as the unit of cost, and modelling the request as a sum of priced components, makes the expensive path visible before it ships.

Mental model

Token economics is a taxi meter with two rates: distance and waiting time. The comparison breaks because the meter also charges for the route you describe at the start, and describing it twice costs as much as driving part of it.

How it works

Mechanism, step by stepInteractive
  1. 1Instrument every call with input, cached input, output and reasoning token counts, plus the model id.
  2. 2Compute cost per request and per feature from those counts, using current provider rates in one configuration file.
  3. 3Attribute cost to a route or feature, so a dashboard shows which product surface spends the budget.
  4. 4Model the marginal cost of a change, such as a longer prompt, an extra retry or an extra tool step, before shipping it.
  5. 5Reduce in order of leverage: shorter prompts, caching stable prefixes, fewer retries, smaller models on easy routes.
  6. 6Set per-request and per-tenant caps, and alert when spend per successful request drifts.

Step 1 of 6

Instrument every call with input, cached input, output and reasoning token counts, plus the model id.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Instrument every call with input, cached input, output and reasoning token counts, plus t… 2. Compute cost per request and per feature from those counts, using current provider rates … 3. Attribute cost to a route or feature, so a dashboard shows which product surface spends t… 4. Model the marginal cost of a change, such as a longer prompt, an extra retry or an extra … 5. Reduce in order of leverage: shorter prompts, caching stable prefixes, fewer retries, sma… 6. Set per-request and per-tenant caps, and alert when spend per successful request drifts.

  1. Instrument every call with input, cached input, output and reasoning token counts, plus the model id.
  2. Compute cost per request and per feature from those counts, using current provider rates in one configuration file.
  3. Attribute cost to a route or feature, so a dashboard shows which product surface spends the budget.
  4. Model the marginal cost of a change, such as a longer prompt, an extra retry or an extra tool step, before shipping it.
  5. Reduce in order of leverage: shorter prompts, caching stable prefixes, fewer retries, smaller models on easy routes.
  6. Set per-request and per-tenant caps, and alert when spend per successful request drifts.

Trade-offs

ChoiceGood forCostFails when
Long prompt with full contextBetter answers on complex tasksInput cost and prefill latencyThe same prefix is resent without caching
Aggressive truncationCheap and fastMissing evidence and worse answersThe task needs long-range context
Small model for most trafficLow cost per requestQuality drop on hard casesYou cannot route by difficulty
Cascade to a large modelCost control with a quality floorTwo calls for escalated requestsEscalation is triggered too often
Trade-off mapInteractive

Good for

Better answers on complex tasks

Cost

Input cost and prefill latency

Fails when

The same prefix is resent without caching

  1. The same prefix is resent without caching
  2. The task needs long-range context
  3. You cannot route by difficulty
  4. Escalation is triggered too often

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Long prompt with full context is good for Better answers on complex tasks, costs Input cost and prefill latency, and fails when The same prefix is resent without caching. Aggressive truncation is good for Cheap and fast, costs Missing evidence and worse answers, and fails when The task needs long-range context. Small model for most traffic is good for Low cost per request, costs Quality drop on hard cases, and fails when You cannot route by difficulty. Cascade to a large model is good for Cost control with a quality floor, costs Two calls for escalated requests, and fails when Escalation is triggered too often

Failure modes

  • Unattributed spend. The bill rises and no one can say which feature caused it. Debug: group cost by route and model for one day and compare with the previous week.
  • Retry multiplier. A transient error doubles cost because each attempt is billed. Debug: count provider calls per user request and compare with successful responses.
  • Cache-blind accounting. Cached input is priced as fresh input and the saving never appears. Debug: reconcile usage fields against the invoice and check cached token counts are recorded.

In production

Alert on cost per successful request and on the share of spend from retries and tool loops. Recalculate unit economics when prices or model routing change, not only at planning time.

Common misconceptions

People think: Cost is proportional to the number of users.

Actually: It is proportional to tokens per request and requests per user, so a prompt change can move it more than growth.

People think: Output length is a minor cost.

Actually: Output tokens are generated serially, priced higher in most APIs, and drive latency, so they are usually the dominant term.

Interview angle

Build the cost model from tokens per component, then show how caching, retries and routing change it. Strong answers attribute spend to features and name the dominant term. Weak answers quote a price per million tokens without saying how many tokens a request actually uses.

Sources