Online-Mind2Web

Online-Mind2Web is the live-web extension of the Mind2Web benchmark, introduced to evaluate web agents on real, public websites rather than on static recordings. The original Mind2Web, published in 2023 by researchers at Ohio State University, contained more than 2,000 human-written tasks across 137 sites but ran against cached snapshots, which limited its realism. Online-Mind2Web answers a simple question: how does an agent perform when the site can change between runs, ads load, content moves, and the network behaves like the real internet?

The benchmark works by sending agents to live URLs and asking them to complete tasks – book a flight, find a product, file a help-desk ticket – with success judged by a human evaluator or an LLM grader. Because the websites are real, agents have to handle pop-ups, A/B tests, login walls, and geolocation differences that synthetic environments hide. Reported scores on Online-Mind2Web are typically lower than on the static Mind2Web or on WebArena, which reflects the additional difficulty of the live web rather than a regression in agent capability.

For AI builders, Online-Mind2Web is the closest publicly available proxy for production reliability of a browser agent. Vendors that publish numbers on this benchmark – or on comparable live-site evaluations such as ClawBench – are signaling that their systems work outside controlled environments. Builders evaluating browser-agent platforms should ask for live-site numbers, not just static-benchmark numbers, and should run their own evaluations on the specific sites that matter for their product.