What is Mind2Web?
Mind2Web is a benchmark for evaluating generalist web agents. It contains 2,350 crowdsourced tasks spanning 137 real websites across 31 domains, with recorded action sequences for each task. Unlike single-site benchmarks, it measures whether an agent transfers to sites and domains it was not trained on. It came out of the OSU NLP Group and was presented at NeurIPS 2023.
Is Mind2Web free and open source?
Yes. Mind2Web is free and open source. The dataset is published on GitHub and Hugging Face under a Creative Commons Attribution 4.0 International license (the code is MIT), so you can use and modify it with attribution. There is no paid tier or commercial plan. As an academic benchmark rather than a hosted product, the only cost is the compute you spend running your own agent against the tasks.
What is the difference between Mind2Web and Online-Mind2Web?
The original Mind2Web evaluates against snapshotted DOMs, which is reproducible but goes stale as sites change. Online-Mind2Web, released in 2025, runs a curated set of tasks against the live web, so it tests real interaction with dynamic pages. It is harder than the static version and gives more honest transfer numbers. Use Online-Mind2Web when you want current results on real websites.
How does Mind2Web compare to WebVoyager?
WebVoyager is largely saturated. Frontier models clear 97% or more on it, so it no longer separates strong agents from weak ones. Mind2Web, especially the Online-Mind2Web variant, still shows a wide spread: many frontier models land in the 30-60% range, even as the best scaffolded systems have climbed into the 90s. That spread is the point. If you want a benchmark that still shows daylight between general-purpose web agents, Mind2Web is the more discriminating choice.
When should I use Mind2Web versus WebArena or ClawBench?
Use Mind2Web when you ship a general-purpose web agent and care whether it generalizes across unfamiliar sites and domains. For domain-specific agents, pair it with WebArena's task subsets, which run in controlled sandboxed environments. ClawBench targets a narrower scraping evaluation. Mind2Web is the breadth-of-transfer test, while the others probe depth within a fixed environment or vertical.
Can Mind2Web be self-hosted?
Mind2Web is not a hosted service you self-deploy. It is a dataset plus evaluation code that you download and run in your own environment, so the data and harness already live wherever you put them. The original tasks run against cached page snapshots, while Online-Mind2Web requires live internet access to reach the real websites it tests. There is no managed cloud version to opt out of.
