Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 37 of 37

Computer use

0 of 37 complete

Why it matters

Much of the software a business runs has no API, or an API that exposes a fraction of the interface. Driving the interface is the only general path, but it replaces a typed contract with perception. The engineering problem is to make an unreliable perception loop safe and debuggable: bounded actions, explicit verification, and a record of exactly what the model saw.

Mental model

Computer use is teaching by screen share: the agent watches what you watch and moves the same mouse. The metaphor breaks because a human shares context and judgement continuously, while the agent gets a still frame and a coordinate with no memory of the last action unless you record it.

How it works

Observe, act, verify: an unchanged frame is the failureInteractive

Keyed transitions

  1. Ground → Act: target resolved
  2. Verify → Halt: same hash ×3 ∨ 40 steps
transitionfailurerepeat

Event · step 1 of 8

Capture · frame 1

Entry: a screenshot plus the accessibility tree, pinned to a 1280 by 800 viewport. Nothing has been clicked. The frame is the only evidence the run has, and it is stored with the action that follows.

In state

  • Capture

Next: Ground resolves a target

Frames
1
Actions
0
Repeated
0
Step 1/8 · Capture · frame 1

Figure. Seven states. Ground turns a frame into one target and Verify compares the next frame instead of trusting the click's return value. An unchanged frame is No-op, whose retry edge is the loop the page warns about; only the frame-hash guard at Halt ends it, and the step cap is the second exit.

What to notice. The frame hash, the 40-step and 40-frame caps, the 1280 by 800 viewport and node 41 are illustrative. Text entry, drag, multi-window applications and native permission dialogs are omitted; the coordinate path shares Ground's guard, and the No-op state covers any action that returns without changing the screen. A click that changes the frame is evidence of an effect, not of the intended effect.

Text version

A state machine with seven states. Capture is the start, a pill. Capture goes to Ground, a hexagonal decision, on frame. Ground goes to Act when the target is resolved. Act goes to Verify on a new frame. Verify has a repeat edge back to Capture labelled changed, a normal edge to Done labelled predicate, a red dashed failure edge to No-op labelled unchanged, and a keyed red dashed edge to Halt labelled same hash times three or 40 steps. No-op has a repeat edge back to Act labelled retry. Done is a terminal pill and Halt is a red failure box. The trace walks Capture, Ground, Act, Verify, Capture, a second Verify that finds the frame unchanged, No-op, a retried click, then Halt on the third identical frame. Counters show frames rising from 1 to 4, actions from 0 to 2, and repeated frames reaching 3.

  1. Capture the observation as a screenshot plus an accessibility or DOM tree, and store both with the action.
  2. Constrain actions to a small vocabulary of click, type, key, scroll and wait, with coordinates or element ids.
  3. After each action take a new observation and check the expected change before proceeding.
  4. Bound the loop with step and time budgets, and stop on repeated identical screens.
  5. Run inside a sandboxed VM with no host credentials and an allowlisted network.
  6. Log every frame and action pair so a failed run can be replayed frame by frame.

Trade-offs

ChoiceGood forCostFails when
Screenshot and coordinatesWorks on any rendered interfaceBrittle to layout, scale and theme changesThe UI changes often or runs at different resolutions
Accessibility treeSemantic targets, more stableMissing or poor in many applicationsThe app is a canvas or remote desktop
Headless browserFast, scriptable, inspectableOnly covers web surfacesThe workflow spans native applications
Human approval before commitContains irreversible mistakesSlower and costlyVolume is high and steps are routine
Trade-off mapInteractive

Good for

Works on any rendered interface

Cost

Brittle to layout, scale and theme changes

Fails when

The UI changes often or runs at different resolutions

  1. The UI changes often or runs at different resolutions
  2. The app is a canvas or remote desktop
  3. The workflow spans native applications
  4. Volume is high and steps are routine

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Screenshot and coordinates is good for Works on any rendered interface, costs Brittle to layout, scale and theme changes, and fails when The UI changes often or runs at different resolutions. Accessibility tree is good for Semantic targets, more stable, costs Missing or poor in many applications, and fails when The app is a canvas or remote desktop. Headless browser is good for Fast, scriptable, inspectable, costs Only covers web surfaces, and fails when The workflow spans native applications. Human approval before commit is good for Contains irreversible mistakes, costs Slower and costly, and fails when Volume is high and steps are routine

Failure modes

  • Silent no-op click. The agent believes it submitted a form that never changed. Debug: assert a post-condition from the next observation after every commit-like action.
  • Coordinate drift. The click lands on the wrong control after a resolution or zoom change. Debug: store the viewport size with each frame and pin it in the sandbox.
  • Loop on a modal. The agent retries the same action because a dialog blocks progress. Debug: hash consecutive screenshots and halt when the same frame repeats.

In production

Track steps per task, repeated-frame rate and action failure rate. Cap wall-clock per task and review recorded runs for a sample of tasks weekly, not only the failures.

Common misconceptions

People think: Computer use replaces integrations.

Actually: It is the fallback for surfaces without an API; an API is faster, cheaper and far more reliable when it exists.

People think: A vision model sees the screen as a human does.

Actually: It sees a sampled image with no reliable notion of focus, hover or what changed since the previous frame.

Interview angle

Strong answers describe the observation format, the action vocabulary, the verification step and the sandbox. They name the failure modes of perception and how a run is replayed. Weak answers say the agent takes screenshots and clicks, with no post-condition checks or isolation.

Sources