Artificial Analysis benchmarked AI search APIs: what the Search Index does and doesn't tell you

Artificial Analysis launched its Search Index on August 18, 2026, the first neutral third-party AI search API benchmark. It scores 11 provider configurations across seven providers on a 0-100 scale by holding one answer model fixed and swapping only the search provider across three research and fact-finding tasks.

For anyone building a RAG pipeline or an agent, this is the number that was missing. Until now, the only comparisons of Exa, Tavily, Parallel, Firecrawl, and their peers came from the vendors themselves, each drawn to a test where its own product wins. A group that spent two years turning model benchmarks into a shared reference point has now aimed the same method at search. That is worth understanding, and it is worth reading with some care, because the headline ranking is the least useful thing in it.

What Artificial Analysis actually shipped

The Search Index announcement covers 11 provider results across seven search providers, among them Parallel, Exa, Firecrawl, You.com, Tavily, and Keenable. Several providers appear more than once because they ship distinct modes, and a mode is really a separate product: Parallel's basic and advanced tiers, or Exa's auto configuration, behave differently enough that averaging them would hide the thing you are choosing between.

Each provider gets a single 0-100 score, which is the equal-weighted mean of three component benchmarks. That score is the part people will screenshot. It is also the part that rewards the least attention, for reasons the rest of this piece works through. If you want the qualitative version of the same layer, our head-to-head of the AI search APIs walks the providers one at a time; the Index adds a measured axis on top of that.

How the provider-swap design works (and why it's the smart part)

The benchmark methodology is a provider-swap. Every task is answered by the same fixed model, GPT-5.6 Luna at medium settings, temperature 0.6, medium reasoning effort. The only thing that changes between runs is which search provider feeds that model. So the score is not "how good is this provider's own answer." It is "how much did this provider's results improve a fixed model's answer against a fixed task." That isolates the search contribution from model variance, which is the most common way vendor comparisons mislead: swap the model, watch the answer improve, then credit the search tool.

The three components test different things. DeepSearchQA is 900 broad research questions that need many searches and resolve to list-style answers, graded by F1 so both misses and spurious additions cost you. BrowseComp is a 200-task subset of hard-to-find facts that need multi-hop browsing, scored on exact-answer accuracy. AA-Omniscience is 600 private factual questions balanced across six domains, and it lets the model abstain, so a confident wrong answer is penalized rather than rewarded. Artificial Analysis also says it filtered known contamination sources out of the search results before grading, which matters because a benchmark answer sitting in the retrieved page is not retrieval, it is a leak.

None of this makes the Index the last word. It makes it an honest measurement of one clearly defined thing, which is more than the layer had a week before it shipped.

What the first Index says

At the top: Parallel Search in advanced mode scores 75, Exa Search on auto scores 74, Firecrawl Search scores 73, and Parallel Search in basic mode also scores 73. That is the leaderboard.

Look at the spread. The top four sit inside two points. On a 0-100 scale, across three benchmarks totaling 1,700 tasks, the difference between first and fourth is smaller than the noise most teams would see re-running the suite on a different day. Read literally, the Index does not say Parallel is better than Firecrawl. It says the leading providers are, for this workload, roughly interchangeable on quality.

That is the finding, and it is easy to miss because a ranked table invites you to treat rank one as the answer. A two-point gap is not a verdict. It is a tie with an ordering imposed on it. A tie on quality throws the decision onto the axes the score compresses away, which is where similar-looking providers stop being similar. Two providers can post near-identical scores while running on completely different underlying indexes, which changes freshness, coverage, and what breaks when your traffic shifts.

The three numbers under the score most builders skip

Cost is two lines, not one

Artificial Analysis reports cost per 1,000 tasks as two figures: the search provider's charge and the answer model's charge. Parallel advanced runs $47.93 in search plus $35.58 in model cost. Exa auto runs $65.57 plus $61.58. Firecrawl runs $30.48 plus $44.94.

The model line is the one people forget, and it is not fixed across providers even though the model is. A provider that returns longer or noisier results makes the model read more and search more often, so it drives up the model bill it does not send you. The provider with the cheaper per-call search can end up more expensive per answer. You cannot see that from the search price on a pricing page, which is why the split matters.

Latency is per task, not per call

The Index reports latency per task, not per search call, and the gap is large: Parallel advanced averages 35.9 seconds per task while Keenable in realtime mode averages 15.1 seconds. Artificial Analysis states the reason plainly: "A provider can be fast per call and still contribute more total time, if the model searches more often against it."

Per-call latency is the number vendors publish. Per-task latency is the number your user feels, because an agent may issue five or ten searches to answer one question. A fast call attached to a strategy that calls repeatedly is not a fast experience. If you are building something interactive, this axis can outrank quality outright.

The benchmark's workload is not your workload

DeepSearchQA, BrowseComp, and AA-Omniscience are research and hard-fact tasks: many-hop questions, obscure facts, broad-domain trivia. If your product does deep research, that is a fair proxy. If it looks up current prices, monitors news, or answers narrow questions inside one vertical, the ranking can invert, because the skills that win a multi-hop trivia hunt are not the skills that win a freshness race. The native-search-versus-API evals showed the same thing, with headline scores that were low and tightly bound to the task set. A benchmark measures its own workload faithfully and yours only by luck.

How to use the Index without over-fitting

Use it as a filter, not a decision. The Index is strong evidence that the top cluster is competent and that anything far down the table has something to prove. That alone is worth having, and it is enough to build a shortlist from.

Then stop reading the rank. Weight the two axes the single score hides against your own shape: if you serve interactive traffic, latency per task leads; if you run high volume, put both cost lines in a spreadsheet with your real call pattern, not the vendor's headline price. A two-point quality spread will not survive contact with a 2x swing in cost or latency, so let those decide among the leaders.

Then run your own evaluation. The providers at the top are close enough that the tiebreaker has to come from your queries, not this one. Running the same comparison on your own traffic is the step the Index makes easier, not the step it replaces. It tells you where to point a small eval, so you test three plausible providers on your questions instead of seven on someone else's.

The measured read

A neutral scoreboard for AI search APIs is genuinely new, and it beats a market where every comparison came from a seller. The provider-swap design is honest, the contamination filtering is the right instinct, and the component benchmarks are well chosen for what they cover. Treat the Index as a credible filter and a sanity check on your own results, not as a ranking to buy off the top row. It measures one workload carefully. Whether that workload is yours is the question the leaderboard cannot answer, and the one worth spending your evaluation budget on.