Why it matters
A tool call carries three decisions: whether to call a tool, which one, and with what arguments. Schema validation only checks the third, and a model can pass it while calling the wrong service with a well-formed query. Evaluation has to cover selection and recovery as well, because an agent that ignores an error response and reports success is more dangerous than one that fails loudly.
Mental model
It is ordering at a restaurant: the right dish, the right modifications, and knowing what to do when it is unavailable. The metaphor breaks because the kitchen can return a plausible but wrong dish and the waiter may not notice until the bill arrives.
How it works
One case, four scores. The fixture uses the same tool set and schemas the agent will see, and the harness injects the failure instead of mocking every call to succeed.
Runner
eval harness
Model
agent under test
Tools
sandboxed
{ "caseId": "tool-042",
"tools": [
{ "name": "search_orders",
"inputSchema": {
"type": "object",
"properties": {
"query": { "type": "string" },
"index": { "enum":
["current", "legacy"] } },
"required": ["query"] } },
{ "name": "get_customer",
"inputSchema": {
"type": "object",
"properties": {
"email": { "type": "string" } } } } ],
"inject": { "get_customer": {
"status": 503 } },
"expect": {
"tool": "search_orders",
"arguments": {
"index": "current" },
"recovery": "name the outage" } }
The real tool set, not a mock. One schema
has two valid index values; only one is
right for this case.Step 1: case. Hover, tap or tab to any message to inspect its payload.
Figure. A case scores selection, schema validity, argument correctness and recovery separately on one run. Here the model picks the declared tool with a schema-valid argument that targets the wrong index, then answers an injected 503 with a claim of success: passing the schema and failing the task are both visible in the same exchange.
What to notice. The failure is a silent tool failure: the injected 503 is followed by a completion claim, so recovery scores zero while schema validity scores one. Selection, arguments and recovery are scored per case, not as an aggregate; description sensitivity and cross-case variance are omitted. Case id, timings and the fixture are illustrative.
Text version
Sequence across Runner (the evaluation harness), Model (the agent under test) and Tools (a sandboxed tool set). Step 1, case: Runner to Model a fixture with caseId tool-042, the real tool descriptors for search_orders and get_customer, an injected 503 on get_customer, and an expectation naming search_orders with index current and the recovery 'name the outage'. Step 2, select: Model to Tools search_orders with query refund window and index legacy; the enum allows legacy, so schema validity passes while the expected argument is current. Tools to Model a 200 result with zero matches from the legacy index. Step 3, fault: Model to Tools get_customer with email ada@example.com; Tools to Model an injected 503 Service Unavailable with Retry-After 1 and error upstream_unavailable. Step 4, claim: Model to Runner a final answer saying the refund was issued, although no tool returned a refund and the customer lookup failed. Step 5, score: an in-process Runner row scoring selection 1, schema 1, arguments 0, recovery 0, unnecessary 0 and missing 1. The four scores stay separate, because one number would hide that the call was valid and wrong.
- Build cases with the tool set and schemas the agent will actually see.
- Label the expected tool or acceptable tool sequence, allowing several valid choices.
- Score selection, schema validity and argument correctness against the task separately.
- Inject error responses and check whether the agent recovers or fabricates success.
- Count unnecessary and missing calls per case.
- Re-run when tool descriptions change, since selection depends on their wording.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Exact tool match | Unambiguous | Punishes equally good alternatives | Several tools can solve the task |
| Schema validity only | Cheap and automatic | Says nothing about correctness | Wrong tool with valid arguments passes |
| Mock tools | Fast and deterministic | Hides real latency and errors | Failure handling matters |
| Live tools in a sandbox | Realistic | Flaky and costly | Volume is large |
Good for
Unambiguous
Cost
Punishes equally good alternatives
Fails when
Several tools can solve the task
- Several tools can solve the task
- Wrong tool with valid arguments passes
- Failure handling matters
- Volume is large
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Exact tool match is good for Unambiguous, costs Punishes equally good alternatives, and fails when Several tools can solve the task. Schema validity only is good for Cheap and automatic, costs Says nothing about correctness, and fails when Wrong tool with valid arguments passes. Mock tools is good for Fast and deterministic, costs Hides real latency and errors, and fails when Failure handling matters. Live tools in a sandbox is good for Realistic, costs Flaky and costly, and fails when Volume is large
Failure modes
- Schema-valid wrong call. The agent calls search with a valid query that targets the wrong index. Debug: compare the arguments with the expected entity, not just the schema.
- Silent tool failure. The tool returns an error and the agent reports success anyway. Debug: check the transcript for error payloads followed by a completion claim.
- Description dependence. Selection collapses after a tool description is reworded. Debug: diff the tool definitions between runs and re-run the selection set.
In production
Track selection accuracy, argument errors and recovery rate per tool. Alert when a tool's error rate changes, since it shifts agent behaviour.
Common misconceptions
People think: If the JSON is valid, the tool call is correct.
Actually: Validity is syntax, while correctness is whether the call advances the task.
People think: Mocking tools is enough for evaluation.
Actually: Mocks remove the errors and latency that agents must handle.
Interview angle
Score selection, arguments and recovery separately and use realistic error responses. A strong answer mentions tool-description sensitivity. Validating JSON schemas and calling that accuracy, with no test of the right tool, is the weak answer.
Sources
- Berkeley Function Calling Leaderboard — function-calling accuracy across models and schemas.
- Qin et al., ToolLLM — tool selection and multi-step API use.
- Yao et al., tau-bench — tool-agent-user interaction with rule constraints.