Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 16 of 33

Debug: cost

0 of 33 complete

Why it matters

A cost spike is rarely one bad request. It is a distribution shift: a new feature that sends the full document, a retry loop that doubles calls, a prompt that grew with examples nobody removed. Because billing arrives aggregated, the first job is attribution. Only after you can say which path and which token type grew can you choose between trimming context, routing to a smaller model, caching a prefix or fixing a retry.

Mental model

Cost debugging is a household budget with itemised bills: the total is uninformative until you know which room is using the power. The metaphor breaks because token spend changes with input content, so the same request can cost different amounts on different days.

How it works

Token accounting before model choiceInteractive
startquestionoutcomepath takenruled out
VerdictChoose· recommended hereConditional· with the named costAvoid· the page advises against

Each edge carries the answer read at the fork; the dashed card at the foot is the blind model swap the page advises against, not a rung.

Step 1 of 6

Attribution first

Split spend by feature, model and token type from request-level records. Until you can name the path, every lever is a guess about the invoice.

question

Can you attribute spend to a feature and a model?

Use request-level usage records, not the monthly invoice. Split input and output tokens, because they have different causes and different fixes.

Path walked

  1. The bill doubled; billing is one number
  2. Can you attribute spend to a feature and a model?

Ruled out: Switch to a cheaper model first

Step 1/6 · Attribution first

Figure. Cost debugging is token accounting before it is model choice: attribute the spend, separate calls from tokens, then split input from output. The input path stops on a cached prefix or a trimmed template; the dashed card at the foot is the blind model swap the page advises against.

What to notice. The ladder is the page's attribution logic, not a measured cost model. No price, token count or cache-hit threshold appears because the page names none. Every lever is a quality trade, so the evaluation set decides whether it ships; the figure draws the lever, not the evaluation.

Text version

A decision ladder for debugging a cost spike. The start says the bill doubled and billing is one aggregated number. Rung 1 asks whether spend can be attributed to a feature and a model; no leads to emitting per-request usage first, because attribution precedes every lever. Yes continues to rung 2: is the growth in calls, not tokens? Yes leads to checking retries, tool loops and fallbacks, because spend can triple with no user-visible change. No continues to rung 3: did input tokens grow? No leads to capping the output, conditional because truncated answers and retries are the cost. Yes continues to rung 4: is a repeated prefix most of the input? Yes leads to caching the repeated prefix; no leads to trimming the template, because examples accumulate release by release. A separate dashed card at the foot, marked Avoid, is switching to a cheaper model first: it moves the cost into answer quality, and the page advises against it without attribution and an evaluation.

  1. Attribute spend per feature and per model using request-level usage records, not the monthly invoice.
  2. Split input and output tokens, since they have different causes and different fixes.
  3. Find repeated prefixes across requests and measure how much of the input they represent.
  4. Check retries, tool loops and fallback paths, which multiply calls without appearing in user-facing metrics.
  5. Test the cheapest plausible fix against the evaluation set before rolling it out.
  6. Set a per-request token cap and an alert on cost per successful task.

Trade-offs

ChoiceGood forCostFails when
Shorter contextDirect token reductionLost evidence and lower answer qualityThe task needs the full document
Smaller model for easy trafficLarge savings on simple requestsRouting errors and inconsistent qualityRequests are hard to classify cheaply
Prompt cachingCheap repeated prefixesCache invalidation and provider supportPrefixes change on every request
Aggressive output limitsLower output costTruncated answers and retriesThe answer legitimately needs length
Trade-off mapInteractive

Good for

Direct token reduction

Cost

Lost evidence and lower answer quality

Fails when

The task needs the full document

  1. The task needs the full document
  2. Requests are hard to classify cheaply
  3. Prefixes change on every request
  4. The answer legitimately needs length

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Shorter context is good for Direct token reduction, costs Lost evidence and lower answer quality, and fails when The task needs the full document. Smaller model for easy traffic is good for Large savings on simple requests, costs Routing errors and inconsistent quality, and fails when Requests are hard to classify cheaply. Prompt caching is good for Cheap repeated prefixes, costs Cache invalidation and provider support, and fails when Prefixes change on every request. Aggressive output limits is good for Lower output cost, costs Truncated answers and retries, and fails when The answer legitimately needs length

Failure modes

  • Retry amplification. A timeout triggers three attempts and spend triples with no user-visible change. Debug: compare request count with user request count for the same hour.
  • Context creep. A prompt template accumulates examples and the input grows every release. Debug: log prompt token count by template version and plot it over time.
  • Cache miss storm. A prefix change invalidates the cache and cost jumps. Debug: measure cache hit rate per prefix and correlate with the deploy timeline.

In production

Emit token usage per request with feature and model labels, and alert on cost per successful task rather than total spend. Cap retries and context length so a single request cannot run away.

Common misconceptions

People think: Cost is driven by output tokens.

Actually: In most chat and RAG paths the repeated input dominates, which is why caching and context trimming matter.

People think: A cheaper model is always the right cost fix.

Actually: It trades quality for price; without an evaluation you are moving the cost into answer quality where it is harder to see.

Interview angle

Show attribution first, then the lever and its quality risk, then the measurement that would catch a regression. Weak answers say they would switch to a cheaper model without knowing which path spends the money.

Sources