Computer Use

Computer use is a form of automation in which a model operates a graphical interface the way a person does: it receives screenshots, decides where to click and what to type, and emits synthetic mouse and keyboard events. It has no privileged access to the DOM, no API contract, and no selectors. The rendered pixels are the entire interface, which is what separates it from browser automation built on Playwright or Puppeteer.

That constraint is also the reason the capability exists. DOM automation reads structure, so it fails wherever structure is absent or useless: canvas-rendered applications such as design tools, charting surfaces, and virtualized spreadsheet grids; legacy desktop software with no API and no web front end; remote desktop sessions; pages that render text into images; and pages whose markup is deliberately obfuscated, with class names regenerated on every build specifically to break scrapers. In each of those cases a selector has nothing to attach to, while a screenshot still shows exactly what a person would see.

Where it loses is speed, cost, and compounding reliability. Every step is a full round trip: render, capture the screen, send image tokens to the model, wait for a decision, dispatch the event. A task a Playwright script completes in seconds takes a computer-use agent minutes. Eden AI's 2026 roundup of computer-use APIs puts per-task cost across the Anthropic, Google, and OpenAI models in the range of roughly one cent to twenty cents, and advises budgeting three to ten times the tokens of an equivalent chat call because each step ships a screenshot. Reliability compounds against you: at 95 percent per-step success, only about a third of twenty-step tasks finish cleanly, and twenty steps is a short workflow by the standards of the demos vendors publish.

The benchmark picture as of July 2026 moved faster than most buying cycles. OSWorld measures agents on 369 real desktop tasks across Ubuntu, Windows, and macOS, against a human baseline the authors measured at 72.36 percent. Steel.dev's OSWorld leaderboard listed its top tracked system at 85.4 percent when last updated in late May 2026, and BenchLM's July 2026 OSWorld-Verified table shows the leading models clustered within about two points of each other near 85 percent, which is the shape of a saturated benchmark. The benchmark's authors answered with a harder long-horizon successor, OSWorld 2.0, released in June 2026, on which the best reported system fell to roughly 21 percent. Those OSWorld 2.0 figures come from secondary write-ups rather than an independently verified leaderboard, so treat them as directional.

For a team building on this, the sensible default is to reach for computer use last. Where a DOM exists, drive it: Playwright and Puppeteer for deterministic flows, Stagehand or Browser Use when you want natural-language instructions over real page structure. Reserve pixel-level control for the surfaces those cannot reach, and expect to pay for two things at once, since the cloud browser session from a provider like Browserbase or Steel.dev is billed separately from the model calls that decide what to do inside it.