What does SimpleQA measure?
It measures short-form factual accuracy. Each of its 4,326 questions has a single, verifiable answer that does not change over time, and a model-based grader labels every response correct, incorrect or not attempted. Because it tracks abstentions, it also measures calibration: whether a model declines to answer when it doesn't know. For search APIs, teams use it to test whether the retrieved results let a model find the right fact.
Why do search API vendors all publish SimpleQA scores?
Because the questions were built to make models without search fail, the benchmark shows a large, easy-to-read lift when you add retrieval. It is also quick and cheap to run, since questions and answers are short. That makes it a convenient shared reference. Tavily, Firecrawl, You.com and Keirolabs have all published runs, and OpenAI ships a reference implementation in its simple-evals repository on GitHub.
Can I compare SimpleQA scores across search providers?
Only with care. Published runs differ in the answer model (GPT-4.1, GPT-5.4 and others), the judge, the number of questions, the tool-call limit, whether a page-fetch tool was allowed, and whether the best of several runs was kept. Keirolabs' 95.3% uses a retrieval-only check rather than a judged answer at all. Scores above 90% sit inside that harness noise, so treat a one- or two-point gap as a tie.
Is SimpleQA saturated?
For models with good browsing tools, largely yes. OpenAI said as much when it introduced BrowseComp in April 2025, and leading search APIs now score above 90%. It still separates search setups from no search, and it still catches weak retrieval: in Tavily's harness, a plain Google results page read by GPT-4.1 scored about 82% and Brave about 76%. For harder multi-step research, BrowseComp or Vals AI's web search benchmark are more discriminating.
What is SimpleQA Verified?
A 1,000-prompt version of SimpleQA from Google DeepMind and Google Research, published in September 2025. The authors removed questions with noisy or incorrect labels, cut near-duplicates, balanced topics and reconciled sources. It is on Hugging Face with a leaderboard on Kaggle. Some search vendors, such as Linkup, report against the Verified set, so check which version a score refers to before comparing it with another.
Is SimpleQA free to run?
Yes. The dataset and grader are open, and you can run it on your own machine against any model and search tool. Your only costs are the API calls: the model answering, the search provider, and the grading model. Running all 4,326 questions with a search tool can add up, which is why some vendors report on seeded samples of 1,000 questions instead of the full set.
