Benchmark Saturation

Benchmark saturation is the point at which a benchmark stops separating systems, because the leading entrants all sit close to its ceiling and the gaps between them fall inside measurement noise. The benchmark keeps producing numbers; those numbers stop carrying information about which system is better. MMLU, GSM8K and HellaSwag reached that state well before 2026, which is why harder successors such as MMLU-Pro, GPQA Diamond and Humanity's Last Exam were built to restore headroom.

Saturation usually arrives alongside a second problem: contamination, where test items leak into training data and a model recalls an answer instead of deriving it. The two compound. A benchmark that has been public for two years has been scraped, discussed in tutorials, mirrored into GitHub repositories and swept into training corpora, so a rising score can reflect exposure rather than capability. A February 2026 AI evaluation digest summarized an analysis of 60 LLM benchmarks that found roughly half of them saturated, with benchmark age and test-set size the strongest predictors, and reported that the usual safeguards (private test sets, open-ended answer formats, template variation) showed no reliable protective effect. Benchmark authors respond with verified subsets, date-filtered problem pools and periodic refreshes, all of which buy time rather than fix the underlying dynamic.

This matters when reading a leaderboard because a score is a property of the model and the evaluation setup together, never the model alone. Two rows in the same comparison table can differ in the evaluator (a deterministic end-state check versus an LLM judge), the task subset, the attempt budget (one greedy run versus best-of-k), the scaffold around the model, and whether the system had tool access. Sierra's tau-bench reports pass^k rather than a single-shot success rate precisely because agents that succeed intermittently look much stronger at k=1; a 2026 arXiv preprint analyzing the May 2026 tau-bench leaderboard reported that the top-ranked system changed depending on whether pass@1 or pass^4 was used for the ranking. None of that is visible in a single percentage.

For anyone choosing a browser agent or an extraction vendor, the workable posture is to cite benchmarks rather than rank on them. ClawBench, Mind2Web, WebVoyager and Vals AI Web Search each measure something real, but they use different task sets, different sites, different judges and different step budgets, so a vendor that leads one is not thereby ahead on another. serp.fast publishes no first-party benchmark numbers for exactly this reason: a directory that ran its own scoring would be adding a fifty-first incomparable leaderboard to a field that already has fifty. The number that should decide a purchase comes from a private harness: 50 to 100 tasks drawn from the sites you actually need, graded against known-correct output, run several times per candidate, with cost per successful task recorded next to accuracy.