What does the Vals AI Web Search benchmark measure?
It measures how much a web search tool improves an AI agent's answers, holding the model constant and changing only the retrieval layer. Vals AI runs the same tasks with a provider's native search and with an independent API like Exa, then compares accuracy. Coverage spans two domains: 208 expert-authored legal-research tasks and 450 finance questions modeled on equity-research work. The result is a per-tool accuracy score, not a latency or cost figure.
Is the Vals AI Web Search benchmark free and open source?
The leaderboard is free to view at vals.ai, but the benchmark itself is not open source. Vals AI builds its datasets in-house and keeps them private, partly to keep them out of model training data. You can read the published results and methodology, but you can't download the tasks or reproduce the runs yourself the way you can with an open benchmark like WebArena.
Which search tool wins the Vals AI Web Search benchmark?
In the July 2026 snapshot the strongest result is Claude Fable 5 paired with Exa, at 48.5% overall accuracy, the highest of any model-and-tool combination tested. The advantage is domain-specific: on finance tasks Exa beats native provider search by about 6.5 points with high statistical confidence, while on legal research the tools score about the same. Standings shift as Vals adds models, so check the live leaderboard for current numbers.
Does an independent search API actually beat a model's native search?
By this benchmark, it depends on the task. On data-heavy finance work an independent tool like Exa shows a clear edge, about 6.5 points, concentrated in market analysis and earnings analysis. On legal research the difference isn't statistically significant. The practical read is that a dedicated search API helps most when answers depend on fresh, specific data, and matters less when the model's built-in search already covers the ground.
Who runs Vals AI?
Vals AI is an independent evaluation company that benchmarks frontier models on real-world, domain-specific tasks across finance, law, software, and other fields. It runs its own evaluations and builds many of its benchmarks in-house, which is how it positions itself as a neutral third party rather than a model vendor. The web search benchmark is one entry in a broader suite that also covers legal, finance, and coding.
How is the Vals AI Web Search benchmark different from agent benchmarks like WebArena?
WebArena, WebVoyager, and ClawBench score whether an agent can finish multi-step tasks on websites. The Vals AI Web Search benchmark asks a narrower question: given a fixed model, how much does the search tool it calls improve the accuracy of its answers. It isolates retrieval quality rather than end-to-end task completion, which makes it the more useful reference when you're choosing a search API instead of an agent framework.
