Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 3 of 37

Agent loop

0 of 37 complete

Why it matters

The loop is where an agent's behaviour actually lives. Prompts set intent and tools set the action space, but the loop sets the rhythm: how much context arrives each turn, how errors are surfaced, how often the model is allowed to try again, and what happens when the eighteenth step is no better than the third.

Get the loop right and a mediocre model handles a surprising range of tasks. Get it wrong and a strong model burns a budget repeating one failed call, because nothing in the harness ever told it that the call had already failed twice. This page is the mechanics; stop conditions covers termination policy in more depth.

Mental model

A control loop in the classical sense: read sensors, compute, actuate, read again. The model is the controller, tools are the actuators, and tool results are the sensors.

The analogy breaks at the controller. A PID loop has a fixed transfer function and guaranteed stability properties. A language model's policy changes with everything in its context, including the last three error messages. There is no stability proof, only bounds: a step cap, a spend cap and an external check on the outcome.

How it works

  1. Build the turn input. Take the persistent transcript, apply the context policy (truncate, summarise, or retrieve), and render it in the provider's message format.
  2. Ask for a decision. The model returns text, tool calls, or both. Treat tool calls as the action channel and prose as commentary, not as an instruction to the harness.
  3. Validate before executing. Check the tool name exists, arguments satisfy the schema, and the caller is authorised for the target resource. Rejection is an observation, not a crash.
  4. Execute with a timeout. Run the tool in the sandbox or service that owns it. Capture structured output, not a stringified exception trace, so the model can reason about the failure.
  5. Append the observation. Include the call ID, the result, and whether it was an error, a timeout or a refusal. Keep the payload small enough to survive twenty more turns.
  6. Check the exits. Stop on a final answer, a budget breach, a repeated action against unchanged state, or a policy refusal.
  7. Persist. Write the transcript and a checkpoint so a crash does not restart the task from zero.

The single most consequential design choice is the shape of the transcript. A loop that appends raw tool output grows linearly and then quadratically in cost, because every step re-sends everything before it. A loop that summarises older steps keeps the window stable but can drop the constraint that made the task well-defined. Anthropic's guidance on context engineering makes the same point: the loop is a context management problem with a model attached.

def loop(task, tools, policy, budget):
    state = State(task=task, transcript=[], step=0, seen_actions=set())
    while True:
        prompt = policy.render(state)                 # truncate, summarise, retrieve
        decision = model(prompt, tools=tools.schemas)
 
        if decision.is_final:
            return policy.finish(state, decision)
 
        for call in decision.tool_calls:
            # Include the state version: a re-read after a change is legitimate,
            # not a loop. Only an identical call against unchanged state is a repeat.
            key = (call.name, canonical_json(call.args), state.version)
            if key in state.seen_actions:
                state.note_repeat(call)
                continue
            state.seen_actions.add(key)
            observation = tools.dispatch(call, timeout=policy.tool_timeout)
            state.append(call, observation)          # bounded size, structured
 
        state.step += 1
        if state.step >= budget.max_steps:
            return policy.abort(state, reason="step_budget")
        if state.tokens > budget.max_tokens:
            return policy.abort(state, reason="token_budget")

Three things in that sketch matter more than they look. Repeats are skipped rather than re-executed, which prevents duplicate side effects. Aborts return a structured reason instead of an exception, so the caller can decide whether to escalate. And policy.finish is a function, not a string, because finishing usually needs a verification step.

Every turn is guarded; a repeat is skipped, not re-executedInteractive

Keyed transitions

  1. Validate → Append: reject · repeat key
  2. Model → Gate: no tool calls
transitionfailurerepeat

Event · step 1 of 9

Turn 1 assembled

Entry: the context policy renders the transcript into a 9.4k-token prompt. Exit is the model call; nothing has been spent on a tool yet.

In state

  • Assemble

Next: Model proposes

Step
Run 1 · 1/12
Tool calls
0
Repeats skipped
0
Step 1/9 · Turn 1 assembled

Figure. Eight states. The repeat key is the triple of tool name, canonical arguments and state version, so a re-read after a change is progress and an identical call against unchanged state is not. The gate owns the exits; the model's claim of completion is an input to it, not the decision.

What to notice. The model proposes; validate, dispatch, append and the gate are harness work. A repeat is recorded as a skipped call rather than dispatched again, which removes the duplicate side effect. The 12-step and 60k-token budgets, the 9.4k prompt, the 10-second tool timeout and the three-repeat window are illustrative, and the context policy is drawn as one state rather than as truncation, summarisation and retrieval separately. A policy refusal exit is omitted; it shares the Halt state.

Text version

A state machine with eight states. Assemble goes to Model on render; Model goes to Validate on a tool call and to Gate on no tool calls. Validate goes to Dispatch on schema and auth, and to Append on a reject or a repeat key. Dispatch goes to Append on result; Append goes to Gate on observation. Gate returns to Assemble on continue, exits to Answer when the predicate holds, and exits to Halt on budget or three repeats. The trace walks Assemble, Model, Validate, Dispatch, Append, Gate, then the continue edge back to Assemble and Model, then Answer, then a skipped duplicate in a second run, then Halt. Counters show the step budget per run, executed tool calls and skipped repeats.

Trade-offs

Loop designGood forCostFails when
Single turn with toolsLookups, one-hop questionsNo recovery from a bad callThe task needs two dependent actions
Interleaved reasoning and actingMost tool tasksOne model call per stepSteps are cheap to batch and the model serialises them
Plan then executeLong tasks with stable environmentsReplanning is bolted on badlyThe environment changes mid-task
Batch parallel calls per turnIndependent reads, searchesHarder to attribute errorsCalls have side effects or ordering
Summarise every k stepsVery long sessionsLossy; can drop constraintsThe task hinges on an early detail

Failure modes

  • No progress, no detection. The agent alternates between two failing strategies until the budget ends. Debug: log a hash of tool name plus normalised arguments per step and count repeats; the trace should show a flat sequence before the abort.
  • Observation poisoning. A tool returns a 4,000-line stack trace, which crowds out the task for the remaining turns. Debug: measure bytes per observation, p95 and max; cap at the tool boundary and return a path to the full artefact.
  • Constraint eviction. Truncation removes the original instruction, and the agent starts optimising the wrong objective. Debug: print the exact prompt at step ten; if the constraints are missing, pin them outside the truncated region.
  • Lost completion signal. The model keeps calling tools because the harness never presents the result of the last action in a form it recognises as success. Debug: replay the final three turns and check whether the observation says what changed, not just that a command exited zero.

In production

Latency per task is roughly the sum of model turns plus tool time, so a thirty-step p95 run at two seconds per model call is already a minute before tools. Streaming the intermediate steps to the user buys perceived speed but demands a UI that can represent partial work honestly.

Cost control belongs in the loop, not the invoice review. Set three caps (steps, tokens, wall clock), return a distinct error for each, and make the caps visible to the caller so it can decide between retry and escalation. Record the abort reason on every run: a rising step-budget rate usually means a tool changed behaviour, not that the model got worse.

Common misconceptions

People think: The loop is a while-loop around an API call, so there is nothing to design.

Actually: The loop owns context policy, error semantics, duplicate suppression, checkpoints and termination. Those five decisions determine reliability more than prompt wording does.

People think: A high step limit makes the agent more capable.

Actually: A high limit mostly makes failures more expensive. Depth should be justified by a task class, and the no-progress detector should fire long before the cap.

Interview angle

Expect to be asked to sketch the loop on a whiteboard and then defend the exits. Strong answers separate model-declared completion from harness-enforced termination, and explain why the harness must not trust the model's claim of success. Then name the context problem: every turn re-sends history, so the loop is also an eviction policy. See agents interview questions, stop conditions and context engineering.

Sources