Why it matters
Much of the software a business runs has no API, or an API that exposes a fraction of the interface. Driving the interface is the only general path, but it replaces a typed contract with perception. The engineering problem is to make an unreliable perception loop safe and debuggable: bounded actions, explicit verification, and a record of exactly what the model saw.
Mental model
Computer use is teaching by screen share: the agent watches what you watch and moves the same mouse. The metaphor breaks because a human shares context and judgement continuously, while the agent gets a still frame and a coordinate with no memory of the last action unless you record it.
How it works
Keyed transitions
- Ground → Act: target resolved
- Verify → Halt: same hash ×3 ∨ 40 steps
Event · step 1 of 8
Capture · frame 1
Entry: a screenshot plus the accessibility tree, pinned to a 1280 by 800 viewport. Nothing has been clicked. The frame is the only evidence the run has, and it is stored with the action that follows.
In state
- Capture
Next: Ground resolves a target
- Frames
- 1
- Actions
- 0
- Repeated
- 0
Figure. Seven states. Ground turns a frame into one target and Verify compares the next frame instead of trusting the click's return value. An unchanged frame is No-op, whose retry edge is the loop the page warns about; only the frame-hash guard at Halt ends it, and the step cap is the second exit.
What to notice. The frame hash, the 40-step and 40-frame caps, the 1280 by 800 viewport and node 41 are illustrative. Text entry, drag, multi-window applications and native permission dialogs are omitted; the coordinate path shares Ground's guard, and the No-op state covers any action that returns without changing the screen. A click that changes the frame is evidence of an effect, not of the intended effect.
Text version
A state machine with seven states. Capture is the start, a pill. Capture goes to Ground, a hexagonal decision, on frame. Ground goes to Act when the target is resolved. Act goes to Verify on a new frame. Verify has a repeat edge back to Capture labelled changed, a normal edge to Done labelled predicate, a red dashed failure edge to No-op labelled unchanged, and a keyed red dashed edge to Halt labelled same hash times three or 40 steps. No-op has a repeat edge back to Act labelled retry. Done is a terminal pill and Halt is a red failure box. The trace walks Capture, Ground, Act, Verify, Capture, a second Verify that finds the frame unchanged, No-op, a retried click, then Halt on the third identical frame. Counters show frames rising from 1 to 4, actions from 0 to 2, and repeated frames reaching 3.
- Capture the observation as a screenshot plus an accessibility or DOM tree, and store both with the action.
- Constrain actions to a small vocabulary of click, type, key, scroll and wait, with coordinates or element ids.
- After each action take a new observation and check the expected change before proceeding.
- Bound the loop with step and time budgets, and stop on repeated identical screens.
- Run inside a sandboxed VM with no host credentials and an allowlisted network.
- Log every frame and action pair so a failed run can be replayed frame by frame.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Screenshot and coordinates | Works on any rendered interface | Brittle to layout, scale and theme changes | The UI changes often or runs at different resolutions |
| Accessibility tree | Semantic targets, more stable | Missing or poor in many applications | The app is a canvas or remote desktop |
| Headless browser | Fast, scriptable, inspectable | Only covers web surfaces | The workflow spans native applications |
| Human approval before commit | Contains irreversible mistakes | Slower and costly | Volume is high and steps are routine |
Good for
Works on any rendered interface
Cost
Brittle to layout, scale and theme changes
Fails when
The UI changes often or runs at different resolutions
- The UI changes often or runs at different resolutions
- The app is a canvas or remote desktop
- The workflow spans native applications
- Volume is high and steps are routine
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Screenshot and coordinates is good for Works on any rendered interface, costs Brittle to layout, scale and theme changes, and fails when The UI changes often or runs at different resolutions. Accessibility tree is good for Semantic targets, more stable, costs Missing or poor in many applications, and fails when The app is a canvas or remote desktop. Headless browser is good for Fast, scriptable, inspectable, costs Only covers web surfaces, and fails when The workflow spans native applications. Human approval before commit is good for Contains irreversible mistakes, costs Slower and costly, and fails when Volume is high and steps are routine
Failure modes
- Silent no-op click. The agent believes it submitted a form that never changed. Debug: assert a post-condition from the next observation after every commit-like action.
- Coordinate drift. The click lands on the wrong control after a resolution or zoom change. Debug: store the viewport size with each frame and pin it in the sandbox.
- Loop on a modal. The agent retries the same action because a dialog blocks progress. Debug: hash consecutive screenshots and halt when the same frame repeats.
In production
Track steps per task, repeated-frame rate and action failure rate. Cap wall-clock per task and review recorded runs for a sample of tasks weekly, not only the failures.
Common misconceptions
People think: Computer use replaces integrations.
Actually: It is the fallback for surfaces without an API; an API is faster, cheaper and far more reliable when it exists.
People think: A vision model sees the screen as a human does.
Actually: It sees a sampled image with no reliable notion of focus, hover or what changed since the previous frame.
Interview angle
Strong answers describe the observation format, the action vocabulary, the verification step and the sandbox. They name the failure modes of perception and how a run is replayed. Weak answers say the agent takes screenshots and clicks, with no post-condition checks or isolation.
Sources
- Xie et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments — real desktop environments and execution-based evaluation for computer agents.
- Deng et al., Mind2Web: Towards a Generalist Agent for the Web — grounding actions in page elements rather than raw coordinates.
- Yang et al., Set-of-Mark Prompting — marking candidate targets to improve visual grounding.