Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 5 of 37

Tool schemas

0 of 37 complete

Why it matters

The schema is the only thing the model knows about a tool before it uses it. If the description says "search" and the tool actually queries a ticketing system with a required project key, the model will call it with a query string, fail, and try again. That is not a model failure; it is a specification failure.

The same object is the harness's enforcement point. Between the model's proposal and the side effect sits validation: does the name exist, do the arguments satisfy the schema, is this caller allowed to touch this resource. A tight schema turns a class of runtime incidents into a rejected call with a useful message.

Mental model

A form at a government office. The fields, their labels and the instructions in small print determine whether people fill it in correctly. The clerk at the window checks completeness and refuses incomplete forms.

The analogy breaks in two places. The model reads the form and the small print at once, so prose quality changes behaviour more than field ordering does. And unlike a clerk, the harness can also rewrite the form mid-conversation: if you change a tool's schema, every later turn sees the new version, including ones already in progress.

How it works

Two codes, two repairsInteractive

A tool schema is read twice: by the model as a prompt, by the harness as a validation contract. The two failures use different codes because they need different repairs.

Model

reads prose

Harness

validates

Tool

orders

Step 1/5 · reject field
2 · invalid_argumentserror · Harness → Model
{ "ok": false,
  "code": "invalid_arguments",
  "retryable": true,
  "errors": [
    { "path": "$.statuss",
      "error": "additional property not allowed" },
    { "path": "$.status",
      "error": "must be one of open, shipped, cancelled" } ] }

Nothing executed. The message names the path
and the allowed values, which is what lets the
model repair instead of guessing.

Step 1: reject field. Hover, tap or tab to any message to inspect its payload.

Figure. The harness is the enforcement point: invalid_arguments names the failing paths and executes nothing; unknown_tool is decided against the registry, not against the model's description. Both return as observations the model can repair from.

What to notice. The failure is a schema that is too permissive or a tool that no longer exists: additionalProperties false turns the invented statuss field into an error, and the registry rejects lookup_customer before any side effect. Description quality and schema drift between deploys are omitted. Emails, order ids and registry contents are illustrative.

Text version

Sequence across Model, Harness (validation and dispatch) and Tool (orders service). Step 1, reject field: an in-process Model to Harness row proposing search_orders with query ada@example.com, status shipping and an invented statuss field; Harness to Model an invalid_arguments error listing path $.statuss, additional property not allowed, and path $.status, must be one of open, shipped, cancelled, with retryable true and no execution. Step 2, repair: Model to Harness a corrected call with query, status shipped and limit 20. Step 3, execute: Harness to Tool search_orders with the validated arguments; Tool to Harness one match, ord_551 shipped. Step 4, observe: an in-process Harness to Model row carrying the result as data. Step 5, reject name: Model to Harness lookup_customer with email ada@example.com; Harness to Model an unknown_tool error with retryable false and registered tools search_orders and get_customer, never sent to the Tool.

  1. Name the action, not the system. search_orders beats orders_api. Names are tokens the model reasons over and greps in its own context.
  2. Write the description for a competent stranger. State what it does, when to use it, when not to, and what the return contains. One or two sentences plus a note on limits.
  3. Constrain the parameters you can. Enums for closed sets, formats for dates, minimum and maximum for ranges, and required only for arguments that genuinely have no safe default.
  4. Forbid the rest. Set additionalProperties: false so a hallucinated field is an error rather than silently ignored.
  5. Define the result shape. Return one predictable object: a status, the data, and a short message. The model plans against the shape it saw in the description.
  6. Define the error shape. Include a machine-readable code, a human-readable message, and whether a retry could help. Errors are observations, not exceptions.
  7. Version explicitly. Renaming a field is a breaking change for every prompt and trace that references it.
SEARCH_SCHEMA = {
    "name": "search_orders",
    "description": (
        "Search orders by customer email or order id. Returns at most 20 matches "
        "with id, status, total_cents and created_at. Use get_order for full detail."
    ),
    "parameters": {
        "type": "object",
        "properties": {
            "query": {"type": "string", "description": "Email address or order id prefix."},
            "status": {"type": "string", "enum": ["open", "shipped", "cancelled"]},
            "limit": {"type": "integer", "minimum": 1, "maximum": 20, "default": 10},
        },
        "required": ["query"],
        "additionalProperties": False,
    },
}
 
def dispatch(call, registry):
    schema = registry.get(call.name)
    if schema is None:
        return {"ok": False, "code": "unknown_tool", "retryable": False}
    errors = validate(call.args, schema["parameters"])   # JSON Schema validator
    if errors:
        return {"ok": False, "code": "invalid_arguments", "errors": errors, "retryable": True}
    return registry.execute(call.name, call.args)

The validation branch is the important one. Returning invalid_arguments with the specific failing path lets the model repair the call. Raising an exception and logging it does not: the model sees a generic failure and often repeats it.

Trade-offs

Schema choiceGood forCostFails when
Many small, single-purpose toolsPrecise calls, clear errorsMore tokens and more selection errorsThe registry grows past what fits comfortably
One broad tool with a mode fieldSmall registries, related operationsModel must set mode correctlyModes have different argument shapes
Strict schema, additionalProperties falseCatching hallucinated fieldsBreaks clients that add fieldsYou need forward-compatible extensions
Loose schema, validate in codeRapid iterationErrors surface late and vaguelyThe side effect is irreversible
Structured error objectSelf-repair by the modelMore output tokens per failureThe error leaks internals
Trade-off mapInteractive

Good for

Precise calls, clear errors

Cost

More tokens and more selection errors

Fails when

The registry grows past what fits comfortably

  1. The registry grows past what fits comfortably
  2. Modes have different argument shapes
  3. You need forward-compatible extensions
  4. The side effect is irreversible
  5. The error leaks internals

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Many small, single-purpose tools is good for Precise calls, clear errors, costs More tokens and more selection errors, and fails when The registry grows past what fits comfortably. One broad tool with a mode field is good for Small registries, related operations, costs Model must set mode correctly, and fails when Modes have different argument shapes. Strict schema, additionalProperties false is good for Catching hallucinated fields, costs Breaks clients that add fields, and fails when You need forward-compatible extensions. Loose schema, validate in code is good for Rapid iteration, costs Errors surface late and vaguely, and fails when The side effect is irreversible. Structured error object is good for Self-repair by the model, costs More output tokens per failure, and fails when The error leaks internals

Failure modes

  • Two tools that sound the same. get_customer and lookup_customer both exist, and calls split unpredictably. Debug: cluster tool calls by name and inspect the ones with high error rates; if two names share a description, merge them.
  • Schema drift. The implementation adds a required field, the schema does not, and every call fails validation. Debug: generate schemas from the handler signatures and assert equality in CI, rather than maintaining two copies.
  • Over-permissive additionalProperties. A typo in a field name is ignored, and the tool runs with a default the model did not intend. Debug: log unvalidated keys at the boundary and fail closed in non-production.
  • Description that describes the implementation. "Calls the v2 endpoint with a bearer token" tells the model nothing about when to use it. Debug: give the description to someone who has not read the code and ask them to choose between two tools; if they hesitate, rewrite it.

In production

Treat the tool registry as a public API with a release process. Version it, diff it between deploys, and keep the old version callable while prompts and evaluations catch up. Log the exact schema hash with every trace so a regression can be attributed to a schema change rather than a model change.

Measure selection accuracy separately from execution accuracy: wrong tool, right tool with wrong arguments, right call with a failed effect. Those three need different fixes, and a single "tool error rate" hides which one is moving. Token cost of definitions belongs in the per-turn cost model, not in a footnote.

Common misconceptions

People think: The JSON Schema is the contract; the description is documentation.

Actually: For the model, the description is the contract. Most wrong calls come from ambiguous prose, not from violated types. Types catch malformed calls, not misconceived ones.

People think: More tools make the agent more capable.

Actually: Each tool adds selection ambiguity, tokens and a failure path. Capability usually improves when overlapping tools are removed.

Interview angle

A common prompt is "design the tool interface for an agent that manages orders". The strong answer starts with the action vocabulary (search, read, mutate), then the schema for each, then the error contract, then the permission model. Interviewers listen for whether you treat descriptions as prompts and errors as observations. A good follow-up answer explains why you would not expose a generic SQL tool to an agent. See agents interview questions, tool use and tool errors.

Sources