How to evaluate a web search API for RAG: run your own bake-off

Nobody can benchmark this for you. A published search API comparison measures whichever queries its author chose, scored by whichever judge its author trusted, on whichever date the author ran it. Your RAG pipeline fails on a specific and unglamorous subset of your traffic: the entity that shares a name with a movie, the question whose answer changed last Tuesday, the internal jargon term that returns nothing useful anywhere. No leaderboard covers that.

The gap is visible in the best public work available. Proxyway's search API report, published 4 March 2026, benchmarked 15 vendors including Brave, Exa, Jina, Linkup, Perplexity, Serper, SerpApi and Tavily on speed and price per 1,000 requests. On result quality the authors wrote that "another important criterion is data relevance; but, as we'll be comparing both direct and indirect representations of the search page, we have no good way to evaluate it." That is an honest methodological limit, and it removes the one variable that decides whether your RAG answers are right.

So build the measurement yourself. The work is a week, not a quarter: harvest 50 to 100 queries from real traffic, define what a supporting result looks like, run every candidate through one harness with everything but the vendor held constant, and price the result in cost per resolved answer. serp.fast publishes no first-party benchmark numbers, deliberately, because the numbers that would matter to you are the ones from your own harness.

Why a published benchmark cannot answer your question

Three of the strongest public efforts as of July 2026 illustrate how far apart benchmark designs sit, and why none of them substitutes for your own run.

Proxyway's March 2026 report ran three separate speed tests of 2,000 requests each at one request per second from a US East Coast server, reporting P50 and P95 with randomly generated queries. That is a clean latency methodology and a useless relevance methodology, which the report says itself.

AIMultiple's agentic search comparison, published May 2026, took the opposite approach: 100 real AI and LLM queries across six categories (research, factual verification, technical documentation, real-time events, comparative, tool discovery), roughly 4,000 retrieved results, each scored by GPT-5.2 for relevance and quality. Its conclusion was that the top four of the eight APIs tested were statistically indistinguishable, with latency spanning 669ms to 13.6 seconds as the real differentiator. A tie at the top is a genuinely useful finding, and it is also a finding that hands the decision back to you.

Vals AI's Web Search Index, published July 2026, isolates the search tool rather than the model: each model runs legal-research and finance-analysis tasks with its native provider search and then again with an independent web-search tool, so the delta is attributable to retrieval. Vals reports that swapping in Exa improved overall accuracy across all four models tested, by roughly 1.5 to 4.3 points. That is the right experimental shape, and the tasks are legal and finance research. If you are building customer-support retrieval over documentation, the transfer is unproven.

None of these benchmarks is wrong. Each one is accurate about the queries it ran, and query distribution dominates that result. A vendor that indexes technical documentation deeply will win a technical-documentation benchmark and lose a local-news benchmark, and neither result tells you what happens on your mix. Use public benchmarks to cut a field of twelve down to four. Do not use them to pick one. The Vals AI Web Search benchmark is worth tracking as a prior; it is not worth deploying against.

Building a golden set of 50 to 100 queries from real traffic

The golden set carries most of the cost of this exercise and most of its value. Once it exists, the rest of the bake-off is arithmetic you can automate.

Sample from logs, not from imagination. Pull the last 30 to 90 days of production queries. If your product is pre-launch, the substitutes are design-partner interview transcripts, support tickets, and the queries your internal team types while dogfooding. Invented queries skew clean and short; real ones are messy, underspecified, and full of proper nouns.

Stratify to match your traffic. Bucket by intent and sample proportionally. A workable set of buckets: factual lookup, entity disambiguation, multi-hop research, time-sensitive, comparative, and long-tail jargon. If 8% of your traffic is time-sensitive, roughly 8% of the golden set should be, because a vendor with a stale index will pass a benchmark that has no time-sensitive queries in it.

Oversample your current failures. Take 10 to 15 queries where your existing pipeline demonstrably returns nothing useful and put them in deliberately. These are the queries that justify a migration. If a challenger vendor fixes eight of them and the incumbent fixes two, that is the strongest single signal the bake-off will produce.

Write down the answer, not just the query. For each query, record what a correct grounded answer contains, and at least one URL you know supports it. This reference answer is what turns a subjective "were the results good" into a checkable "did the returned set contain the evidence." It is the slowest part of the setup and the part that makes every later measurement cheap.

Freeze it and version it. Commit the set to your repo with a date. When a vendor ships a new mode in October and you re-run, you need to know you are running the same 87 queries you ran in July. A golden set that quietly drifts produces trend lines that mean nothing.

Deciding what to measure

Four metrics cover a RAG retrieval decision. More than four and the labeling burden grows without improving the decision.

Answer-support rate is the metric closest to user experience and the most expensive to compute. Recall@k is its cheap proxy. Run recall@k across the full set and answer-support on a 30-query subset, then check that they move together; if they diverge sharply, your recall labels are probably too generous.

Precision deserves less weight than intuition suggests. A reranker or a filter step can discard noise you retrieved. Nothing downstream can recover a document that was never returned. Optimize the pipeline for recall at the retrieval stage and handle precision after, which is also the argument for setting k higher than feels comfortable during evaluation and tuning it down later.

Set k once and keep it. Comparing a vendor at k=10 against another at k=5 is not a comparison.

Judging results: human labels, LLM-as-judge, and how each fails

You have two options for turning a result set into a score, and both are flawed in known ways.

Human labeling is the ground truth and it drifts. A labeler applying a five-point relevance scale on Monday is not the same labeler on Friday, and two labelers disagree more than teams expect. Fix what you can: write a rubric with worked examples for each score, have two people label the same 30 queries independently, and check agreement before labeling the rest. If two humans following your rubric disagree on a third of items, the rubric is the problem, not the labelers. Budget roughly two to four minutes per query per vendor, which is why nobody labels 100 queries across six vendors by hand.

LLM-as-judge is what makes the volume affordable, and it comes with a documented bias profile. The research literature on judge reliability consistently identifies three failure modes: position bias, where the judge favors whichever candidate appeared first or last in the prompt (studied systematically in "Judging the Judges", arXiv 2406.07791); verbosity bias, where longer semantically equivalent output scores higher; and self-preference bias, where a judge rates output from its own model family more favorably (arXiv 2410.21819). All three are directly dangerous here. Vendor A returns longer snippets, so verbosity bias hands it points. You always list vendors in the same order, so position bias hands the first one points. Your judge is made by a company that also sells a search API in your bake-off, and self-preference does the rest.

The mitigations are mechanical:

• Randomize order on every comparison, and run each pair twice with the order flipped. Score only where both runs agree, or average them.
• Strip vendor identity from what the judge sees. No URLs to vendor domains, no field names that give it away, no metadata blocks.
• Normalize length. Truncate every candidate snippet to the same token budget before judging, so a vendor cannot buy a higher score with volume.
• Calibrate against humans. Hand-label 50 items, run the judge on the same 50, and measure agreement. Report the agreement number alongside your results. If you cannot state it, you do not know what your scores mean.
• Do not judge with a model whose vendor is in the bake-off. If you are evaluating an answer-synthesis API from a model provider, use a judge from a different provider.

Holding variables still while vendors change theirs underneath you

Most bake-offs fail here rather than at the metric stage. The retrieval call is one variable in a pipeline with a dozen, and the others move.

Pin all of these and record them with the run: k, the exact query strings, the model and its version, the prompt template, temperature, any domain include or exclude filters, the region you call from, and the date. Then pin the one that catches most teams out: the vendor's retrieval mode.

Vendors sell several products behind one endpoint. Exa exposes Instant, Fast, Auto and Deep modes at different latencies and prices. Tavily exposes a search_depth dial spanning ultra-fast through advanced. Linkup sells Fast, Standard and Deep tiers. Parallel exposes a fast Search API and a Task API with a ladder of processor tiers. Benchmarking Exa Deep against Tavily's basic depth is a pricing comparison wearing a quality comparison's clothes. Decide which latency and price tier you would actually ship, then compare every vendor at that tier.

Content shape is the second silent variable. An index API returns extracted text, a SERP API like Serper.dev returns ranked links, and Brave Search API returns results from its own independent index rather than a scrape of Google's. If a vendor returns links, you must add a fetch-and-extract stage before judging, and the extractor becomes part of the measurement. Use the same extractor for every arm; Jina AI's Reader is a common choice precisely because one normalizer across all arms removes extraction quality as a confound.

Domain coverage is the third. Valyu spans arXiv, PubMed and SEC filings alongside general web. If a fifth of your golden set is biomedical, a vendor with dedicated coverage there will look better than a general benchmark would predict, and that is a real advantage rather than an artifact. The point of running your own set is that this shows up.

Finally, timestamp everything. Vendors reindex, reprice, and ship new modes continuously. A result from April is a historical note, not a current fact.

Measuring latency honestly

Latency is the easiest thing here to measure, and the easiest to measure in a way that flatters every vendor in the test.

Report p50, p95 and p99, not a mean. Means hide the tail, and the tail is what your users experience as "the app is broken." AIMultiple's May 2026 run put the spread across eight APIs at 669ms to 13.6 seconds, which is the difference between an interactive feature and a batch job.

Measure from where you will actually deploy. Proxyway ran from a US East Coast server; if your inference runs in Frankfurt, its numbers describe someone else's network path. Warm up before you record, then also record the cold path separately, because the first call after an idle period is what a low-traffic product hits all day.

Include your retry policy in the measurement. If your client retries once on timeout, the user-visible latency of a retried request is both attempts plus backoff, and that request belongs in the p99 rather than being quietly excluded. Count timeouts and 5xx responses as failures with a recorded failure rate, never as missing data. A vendor with a 2% silent failure rate and a great p50 is worse than a slower vendor that always answers.

Test at your real concurrency. One request per second, as in Proxyway's methodology, is a clean way to isolate per-request latency and tells you nothing about behavior at 50 concurrent calls, where rate limits and queuing appear.

Cost per resolved answer instead of cost per call

Per-call pricing is the number on the pricing page and the wrong unit for a RAG decision. The unit that matters is what it costs to produce one correct, grounded answer.

The arithmetic:

cost per resolved answer =
  (search calls x search price
   + content fetches x content price
   + input tokens x model input price
   + output tokens x model output price)
  / answers your judge scored as supported

Two things fall out of this. First, the denominator is your quality metric, so a vendor with a 20% higher answer-support rate is 20% cheaper per answer at identical list price. Second, a cheap search call that hands you links can be more expensive than a pricier call that hands you content, because you pay for the fetch stage and for the extra tokens of unfiltered text you push into the model.

Run it as a table with your own quoted rates, one row per vendor, and include every line item you would actually be billed for. Both the search call and the content or extract call, if the vendor prices them separately. The token cost of the payload each vendor returns, which is where a verbose vendor loses. The fetch-and-extract cost for any SERP-shaped arm. Retries, which are pure overhead multiplied by your failure rate.

Do not carry vendor pricing from a directory or from an article into your model. Pricing in this category changed repeatedly through the first half of 2026, and published rates in the low single digits of dollars per 1,000 requests are the norm rather than a guarantee. Take the number from the vendor's own pricing page on the day you build the model, and note the date next to it.

Reading third-party benchmarks after you have your own numbers

Public benchmarks get more useful once you have your own numbers to read them against, which is the reverse of the order they usually get read in.

Read them as a prior on questions your own set cannot answer. Your 87 queries will not tell you which vendor holds up at 10x your volume, or how a vendor behaves on query types you do not serve yet but might next year. A public benchmark is decent evidence for both.

Four questions to ask of any benchmark, including the ones cited above:

What was the query distribution? If the report does not publish its queries or at least their category breakdown, treat the ranking as unusable. AIMultiple publishes its six categories and counts, which is why the result is interpretable.

Who judged, and how? An LLM judge without a stated human-agreement number is an unvalidated instrument. Ask whether order was randomized and whether vendor identity was masked.

When was it run? Proxyway tested most vendors mid-January 2026 and Exa mid-February, and published in March. A February measurement read in July describes a product that has shipped several releases since.

Who paid? Vendor-published benchmarks are not worthless, but they are marketing artifacts and their query sets are chosen by an interested party. Linkup promotes a SimpleQA factuality lead; Parallel has claimed BrowseComp advantages over competitors. Treat vendor claims as directional and, where you can, check whether the benchmark is one the vendor selected after the fact.

Where a public result agrees with your own, you have gained confidence. Where it disagrees, the interesting question is which part of your query mix explains the gap, and that question is usually more informative than the ranking either way. Our AI search APIs compared guide is a snapshot of the same shape, and it should be read with the same suspicion.

Turning the bake-off into a quarterly harness

The one-time bake-off answers a procurement question. The harness answers the question you actually have, which is whether the vendor you chose is still the right one.

Keep the golden set in version control alongside the runner, the rubric, and the judge prompt. Every artifact that produced a number should be reproducible from a commit hash.

Split the cadence. Nightly, run 20 queries from the set against your production vendor and alarm on recall@k, answer-support rate, p95 latency, and failure rate. This is cheap and catches the common case, which is not a vendor going down but a vendor quietly getting worse after an index or ranking change. Quarterly, run the full set against your incumbent plus two challengers, and re-price cost per resolved answer with current published rates.

Let the set evolve, slowly and on the record. Add roughly 10 queries per quarter drawn from recent production failures, retire queries whose reference answers have gone stale, and note both in the changelog. A set that never changes stops representing your traffic; a set that changes silently destroys your trend line.

Keep the runner-up integrated. If your retrieval layer sits behind one interface with two implementations, switching vendors is a config change rather than a sprint, which is what lets you put your quarterly numbers on the table in a renewal conversation instead of just filing them.

The short version

Build a golden set of 50 to 100 real queries with reference answers, freeze it, and run every candidate through one harness with k, model, prompt and vendor retrieval mode held constant. Score recall@k and answer-support rate, judge with a rubric whose LLM judge you calibrated against a human-labeled subset, and report p95 alongside failure rate rather than a mean. Divide total pipeline cost by supported answers, not by API calls. Then read the public benchmarks, and note where they disagree with you, because that gap is a fact about your traffic. For the upstream question of which categories of vendor belong in the shortlist at all, start with where RAG data actually comes from.