Why it matters
A cost spike is rarely one bad request. It is a distribution shift: a new feature that sends the full document, a retry loop that doubles calls, a prompt that grew with examples nobody removed. Because billing arrives aggregated, the first job is attribution. Only after you can say which path and which token type grew can you choose between trimming context, routing to a smaller model, caching a prefix or fixing a retry.
Mental model
Cost debugging is a household budget with itemised bills: the total is uninformative until you know which room is using the power. The metaphor breaks because token spend changes with input content, so the same request can cost different amounts on different days.
How it works
Each edge carries the answer read at the fork; the dashed card at the foot is the blind model swap the page advises against, not a rung.
Step 1 of 6
Attribution first
Split spend by feature, model and token type from request-level records. Until you can name the path, every lever is a guess about the invoice.
question
Can you attribute spend to a feature and a model?
Use request-level usage records, not the monthly invoice. Split input and output tokens, because they have different causes and different fixes.
Path walked
- The bill doubled; billing is one number
- Can you attribute spend to a feature and a model?
Ruled out: Switch to a cheaper model first
Figure. Cost debugging is token accounting before it is model choice: attribute the spend, separate calls from tokens, then split input from output. The input path stops on a cached prefix or a trimmed template; the dashed card at the foot is the blind model swap the page advises against.
What to notice. The ladder is the page's attribution logic, not a measured cost model. No price, token count or cache-hit threshold appears because the page names none. Every lever is a quality trade, so the evaluation set decides whether it ships; the figure draws the lever, not the evaluation.
Text version
A decision ladder for debugging a cost spike. The start says the bill doubled and billing is one aggregated number. Rung 1 asks whether spend can be attributed to a feature and a model; no leads to emitting per-request usage first, because attribution precedes every lever. Yes continues to rung 2: is the growth in calls, not tokens? Yes leads to checking retries, tool loops and fallbacks, because spend can triple with no user-visible change. No continues to rung 3: did input tokens grow? No leads to capping the output, conditional because truncated answers and retries are the cost. Yes continues to rung 4: is a repeated prefix most of the input? Yes leads to caching the repeated prefix; no leads to trimming the template, because examples accumulate release by release. A separate dashed card at the foot, marked Avoid, is switching to a cheaper model first: it moves the cost into answer quality, and the page advises against it without attribution and an evaluation.
- Attribute spend per feature and per model using request-level usage records, not the monthly invoice.
- Split input and output tokens, since they have different causes and different fixes.
- Find repeated prefixes across requests and measure how much of the input they represent.
- Check retries, tool loops and fallback paths, which multiply calls without appearing in user-facing metrics.
- Test the cheapest plausible fix against the evaluation set before rolling it out.
- Set a per-request token cap and an alert on cost per successful task.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Shorter context | Direct token reduction | Lost evidence and lower answer quality | The task needs the full document |
| Smaller model for easy traffic | Large savings on simple requests | Routing errors and inconsistent quality | Requests are hard to classify cheaply |
| Prompt caching | Cheap repeated prefixes | Cache invalidation and provider support | Prefixes change on every request |
| Aggressive output limits | Lower output cost | Truncated answers and retries | The answer legitimately needs length |
Good for
Direct token reduction
Cost
Lost evidence and lower answer quality
Fails when
The task needs the full document
- The task needs the full document
- Requests are hard to classify cheaply
- Prefixes change on every request
- The answer legitimately needs length
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Shorter context is good for Direct token reduction, costs Lost evidence and lower answer quality, and fails when The task needs the full document. Smaller model for easy traffic is good for Large savings on simple requests, costs Routing errors and inconsistent quality, and fails when Requests are hard to classify cheaply. Prompt caching is good for Cheap repeated prefixes, costs Cache invalidation and provider support, and fails when Prefixes change on every request. Aggressive output limits is good for Lower output cost, costs Truncated answers and retries, and fails when The answer legitimately needs length
Failure modes
- Retry amplification. A timeout triggers three attempts and spend triples with no user-visible change. Debug: compare request count with user request count for the same hour.
- Context creep. A prompt template accumulates examples and the input grows every release. Debug: log prompt token count by template version and plot it over time.
- Cache miss storm. A prefix change invalidates the cache and cost jumps. Debug: measure cache hit rate per prefix and correlate with the deploy timeline.
In production
Emit token usage per request with feature and model labels, and alert on cost per successful task rather than total spend. Cap retries and context length so a single request cannot run away.
Common misconceptions
People think: Cost is driven by output tokens.
Actually: In most chat and RAG paths the repeated input dominates, which is why caching and context trimming matter.
People think: A cheaper model is always the right cost fix.
Actually: It trades quality for price; without an evaluation you are moving the cost into answer quality where it is harder to see.
Interview angle
Show attribution first, then the lever and its quality risk, then the measurement that would catch a regression. Weak answers say they would switch to a cheaper model without knowing which path spends the money.
Sources
- OpenTelemetry GenAI semantic conventions — standard token usage attributes on model spans.
- Anthropic, Prompt caching — how repeated prefixes are cached and invalidated.
- vLLM documentation — prefix caching and batching effects on serving cost.