Why it matters
A tool can return a correct result that still causes a wrong answer: a two-hundred-row table where the relevant row is last, a document cut mid-sentence, an error page rendered as content. The observation is the interface between deterministic code and a probabilistic reader, so its size, order and framing directly affect accuracy and cost.
Mental model
An observation is a briefing note, not a data dump. The metaphor breaks because a briefer knows the reader's question and can omit safely, while the tool often does not, so filtering must be conservative and every omission visible.
How it works
Keyed transitions
- Cap + marker → Model step: omitted count ∧ expand id
- Fits? → Ambiguous empty: ∅ as list
Event · step 1 of 9
Tool returns 214 rows
Entry: a query returns 214 rows with navigation, headers and a four-line footer. Nothing is shaped yet; the raw payload is 9,800 tokens, and most of it is chrome.
In state
- Tool result
Next: Shape chosen
- Tokens
- 9.8k
- Omitted
- 0
- Marks
- 0
Figure. Nine states. Shape chooses a ranked list, a record or a reference; Rank orders rows before Fits cuts, and Cap is the state where an omission becomes visible. Model step is terminal because it reads a marked, bounded observation. The red states are the page's failures: an empty result shaped like a list, a cut with no marker, and boilerplate left in.
What to notice. The 1,500-token cap, the top-8 cut, the 214 rows and the 208 omitted are illustrative. The full raw payload and the summarised variant share Shape but are not drawn; a summary that rewords ids is a distinct failure the page covers in prose. The reference-and-expand shape shares Model step.
Text version
A state machine with nine states. Tool result is the start, a pill. Tool result goes to Shape, a hexagonal decision, on raw. Shape goes to Rank on ranked list, and to Boilerplate flood, a red failure box, on navigation kept. Rank goes to Fits on top 8. Fits goes to Cap plus marker, a checkpoint, on over cap, to Model step, a terminal pill, when the observation is at most 1.5k tokens, and to Ambiguous empty, a red failure box, when an empty result is shaped as a list. Cap plus marker goes to Model step on a marker with the omitted count, and to Silent cut, a red failure box, on no marker. The trace walks Tool result, Shape, Rank, Fits, Cap plus marker, Model step, then Ambiguous empty, then Silent cut, then Boilerplate flood. Counters show observation tokens falling from 9.8k to 1.4k, omitted rows at 208, and marks at 1 only after Cap.
- Decide the shape per tool: a compact record, a ranked list, or a reference the agent can expand.
- Cap size with an explicit marker stating how much was omitted and how to get the rest.
- Put the most decision-relevant fields first and keep exact identifiers verbatim.
- Strip boilerplate such as navigation, headers and repeated disclaimers before it reaches the model.
- Give empty, error and partial results distinct structures.
- Log the exact observation text with the step so a bad decision can be attributed.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Full raw output | No information loss | Token cost and attention dilution | Outputs are large or repetitive |
| Summarised output | Cheap and focused | Summariser errors and lost ids | Exact values matter |
| Ranked top-k | Relevant and short | Misses the tail | The answer is in an unusual record |
| Reference and expand | Bounded context, drill-down possible | Extra turn and latency | The agent rarely needs the detail |
Good for
No information loss
Cost
Token cost and attention dilution
Fails when
Outputs are large or repetitive
- Outputs are large or repetitive
- Exact values matter
- The answer is in an unusual record
- The agent rarely needs the detail
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Full raw output is good for No information loss, costs Token cost and attention dilution, and fails when Outputs are large or repetitive. Summarised output is good for Cheap and focused, costs Summariser errors and lost ids, and fails when Exact values matter. Ranked top-k is good for Relevant and short, costs Misses the tail, and fails when The answer is in an unusual record. Reference and expand is good for Bounded context, drill-down possible, costs Extra turn and latency, and fails when The agent rarely needs the detail
Failure modes
- Silent truncation. The model treats a cut list as complete and reports a wrong total. Debug: assert every truncated observation carries an explicit marker and a count.
- Boilerplate flood. Navigation text dominates the observation and the answer quotes it. Debug: compare observation length with extracted content length per tool.
- Ambiguous empty. No results and a failed call look identical. Debug: check the observation schema has distinct states for empty, error and partial.
In production
Track observation tokens per tool, truncation rate and the share of observations the agent quotes. Alert when a tool's average observation size jumps after a deploy.
Common misconceptions
People think: Returning everything is the safest choice.
Actually: Unfiltered output costs tokens and hides the relevant line; the safest is bounded output with visible omissions.
People think: Summarising is always safe.
Actually: A summary that drops an id or a number makes a correct tool result unusable.
Interview angle
Strong answers treat the observation as an interface with a budget, describe truncation markers and empty-versus-error states, and explain how observation quality is measured. Weak answers say the tool returns JSON.
Sources
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts — why position and length of supplied context change what is used.
- Anthropic, Tool use with Claude — how tool results are returned into the model's context.