Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 31 of 33

Troubleshoot: token blowup

0 of 33 complete

Why it matters

Token usage is invisible until it hits a limit or an invoice. When it does, the cause is one of a few shapes: conversation history that is never truncated, a tool returning a large payload, retrieved context that grew with k, or a retry path that re-sends the whole prompt. Each has a different fix, and prompt shortening only addresses the first. The drill is to attribute before cutting, because trimming the wrong part reduces quality without reducing cost.

Mental model

It is a suitcase that gets heavier at every stop: each turn adds an item and nobody removes one. The metaphor breaks because some items are added automatically by tools, so the traveller is not the only one packing and the growth is easy to miss until a limit is hit.

How it works

Mechanism, step by stepInteractive
  1. 1Get usage records per request with input, output and cached token counts, and group by conversation.
  2. 2Plot tokens per turn for one long conversation to see whether growth is linear and unbounded.
  3. 3Break input tokens into system prompt, history, retrieved context and tool output.
  4. 4Check retries and fallbacks by comparing model calls with user requests.
  5. 5Cap history by summarising or windowing, cap tool output size, and cap output tokens.
  6. 6Re-measure the same conversation and confirm quality on the evaluation set did not drop.

Step 1 of 6

Get usage records per request with input, output and cached token counts, and group by conversation.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Get usage records per request with input, output and cached token counts, and group by co… 2. Plot tokens per turn for one long conversation to see whether growth is linear and unboun… 3. Break input tokens into system prompt, history, retrieved context and tool output. 4. Check retries and fallbacks by comparing model calls with user requests. 5. Cap history by summarising or windowing, cap tool output size, and cap output tokens. 6. Re-measure the same conversation and confirm quality on the evaluation set did not drop.

  1. Get usage records per request with input, output and cached token counts, and group by conversation.
  2. Plot tokens per turn for one long conversation to see whether growth is linear and unbounded.
  3. Break input tokens into system prompt, history, retrieved context and tool output.
  4. Check retries and fallbacks by comparing model calls with user requests.
  5. Cap history by summarising or windowing, cap tool output size, and cap output tokens.
  6. Re-measure the same conversation and confirm quality on the evaluation set did not drop.

Trade-offs

ChoiceGood forCostFails when
Summarise historyBounded contextLoss of detail and summary errorsThe task depends on exact earlier text
Window the last N turnsSimple and predictableForgets constraints set earlyInstructions were given at the start
Truncate tool outputLarge immediate savingsModel loses evidenceThe answer is inside the payload
Raise the context limitNo quality lossHigher cost and latencyUsage is already near the budget
Trade-off mapInteractive

Good for

Bounded context

Cost

Loss of detail and summary errors

Fails when

The task depends on exact earlier text

  1. The task depends on exact earlier text
  2. Instructions were given at the start
  3. The answer is inside the payload
  4. Usage is already near the budget

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Summarise history is good for Bounded context, costs Loss of detail and summary errors, and fails when The task depends on exact earlier text. Window the last N turns is good for Simple and predictable, costs Forgets constraints set early, and fails when Instructions were given at the start. Truncate tool output is good for Large immediate savings, costs Model loses evidence, and fails when The answer is inside the payload. Raise the context limit is good for No quality loss, costs Higher cost and latency, and fails when Usage is already near the budget

Failure modes

  • History growth. Each turn resends the full transcript and cost rises quadratically. Debug: plot cumulative input tokens against turn number for one conversation.
  • Tool payload. A search tool returns a large document and every later turn carries it. Debug: attribute input tokens by message role and look for one dominant tool message.
  • Retry duplication. A timeout re-sends the prompt and usage doubles per user request. Debug: count model calls per request id and compare with one.

In production

Emit usage per request with a conversation id, and alert on tokens per turn and cost per successful task. Cap history, tool output and output length so a single conversation cannot grow without bound.

Common misconceptions

People think: Longer prompts always improve quality.

Actually: Past the relevant evidence, extra tokens add cost and distract the model from the instruction.

People think: Caching removes the need to trim context.

Actually: Caching discounts repeated prefixes; it does not bound growth within a conversation.

Interview angle

Strong answers attribute usage by component, then cap the component that grew and verify quality. Weak answers say they would shorten the prompt without knowing which part is large.

Sources