The web data API stopped being one primitive: buying a platform, not a product

Two years ago, an AI team assembling web access bought a search API from one vendor, a scraper from another, maybe a headless browser from a third, and wrote the glue between them. In 2026 that shopping list collapsed. The same vendors folded search, scraping, crawling, extraction, a cloud browser and an agent runner behind a single API and one credit meter. Firecrawl, Tavily and Exa each now sell most of the retrieval stack rather than one primitive of it. The consequence for a buyer is easy to state and easy to underrate: you are no longer choosing a search API or a scraper, you are choosing a platform, and platforms are harder to price-compare and harder to leave.

A web data platform for AI agents is a single service that bundles what used to be separate products: web search, page fetching and scraping, site crawling, structured extraction, a hosted browser, and sometimes an autonomous research endpoint, all metered on one billing system. What changed in 2026 is that this bundle became the default unit of sale. It buys simpler integration at the cost of harder price comparison and harder exit.

What actually consolidated in 2026

Look at the endpoint lists and the pattern is hard to miss. Firecrawl exposes search, scrape, parse, crawl, map, an interact step, an /agent endpoint for autonomous multi-step research, and a browser sandbox, all drawn from one credit balance, with a search costing 2 credits per 10 results and paid plans running up to $599 and beyond per month. Tavily covers /search, /extract, /crawl, /map and /research, billed pay-as-you-go at about $0.008 per credit. Exa sells search-with-contents and contents-only as one graded product, around $7 and $1 per thousand respectively. Three vendors, three overlapping bundles, and none of them is selling a single primitive anymore.

The money is moving the same direction. Nebius agreed to acquire Tavily for $275 million in February 2026, folding an agentic search API into a larger AI cloud. Exa raised a $250 million Series C at a $2.2 billion valuation in May 2026, capital that only makes sense if the product is infrastructure rather than a feature. Browser infrastructure is folding into the same stack: Browserbase raised a $40 million Series B at about a $300 million valuation and had run more than 50 million browser sessions across over 1,000 companies by the time it announced. Steel.dev ships the same capability as open source, and Hyperbrowser packages stealth features with its own agent framework. The category that used to be a headless browser you rent is now one layer of a retrieval platform.

The pull is coming from the workload, not just the vendors. An agent doing a research task does not want a ranked list of links it then has to fetch and clean in a second and third call. It wants the content already extracted and shaped for a model, which is why the old search-then-scrape two-hop is being folded into one request. Firecrawl's pitch is that a single call finds the pages and returns their content. Tavily's is that its search hands back model-ready context rather than raw HTML. Once a vendor owns both the finding and the extracting, adding a crawl, a browser and an agent loop is a smaller step than it looks, and the credit meter is what ties them together. The bundle is less a packaging decision than the shape the agent use case asked for.

If you want the older mental model of these pieces as separate tools, our guide to search, crawl and extract APIs still maps the primitives. The point of this post is what changes when one vendor sells you all of them at once.

What bundling actually buys you

The upside is real. One platform means one bill, one API key, one authentication and rate-limit model, and one place to read logs when a job fails. The glue code between "search returned these URLs" and "now fetch and clean each one" disappears into a single call, because the vendor already owns both ends of it. For a small team shipping an agent, that removed integration surface is often worth more than a few cents of margin per thousand requests.

There is a second, less obvious benefit in 2026. Access to the open web is getting gated, and a single fetch-and-crawl path inside one platform is easier to keep working than a patchwork. Cloudflare's default, effective September 15, 2026, blocks mixed-use AI crawlers on ad-supported pages unless a company separates its crawlers by purpose. A vendor that owns search, crawl and browser in one place can manage that identity and fallback logic for you, instead of leaving you to reconcile three vendors' bot reputations. Bundling concentrates the compliance problem in one contract, which is easier to reason about even when it is not cheaper.

What it quietly costs you

The costs are quieter, which is why they are worth writing down.

The first is concentration. When search, extraction and the browser all come from one vendor, a single outage, price change or policy shift lands on your whole retrieval path at once. The thing that made the integration simple, one throat to choke, is also the thing that removes your fallbacks. If that vendor deprecates an endpoint, or gets acquired and re-prices as Tavily's new owner now could, there is no second supplier already carrying part of the load.

The second is pricing that resists comparison. A credit is a clean abstraction for the vendor and a fog for the buyer. When one credit can mean a search, a page fetch, or a full browser render, the sticker price per credit tells you almost nothing about your cost per finished job. Two platforms with similar per-credit rates can differ several times over on the same workload, depending on how each meters rendering, retries and extraction. Working out the real number is the whole subject of our guide to web data cost per answer, and the bundle makes that arithmetic harder, not easier.

The third is exit cost. Retrieval bundled with extraction schemas, crawl state and an agent endpoint does not lift out cleanly. The more of the stack you adopt from one platform, the more of your own code assumes that platform's response shapes, and the higher the wall when you want to move. Web data API lock-in is not a clause in a contract, it is the accumulated shape of the integration.

How to buy a platform without getting captured

None of this argues against buying a platform. It argues for buying one deliberately.

Normalize on your own workload, not the vendor's demo. Take your real mix of search calls, fetches and extractions, run it through the credit model, and compare platforms on cost per finished job rather than per credit. The Exa vs Tavily comparison is the right altitude for that on the search side, and the same method extends across the fuller bundle.

Keep a second retrieval channel warm. Even a thin, tested integration with an alternate provider, used for a slice of traffic, preserves the fallback that bundling removes and gives you a real price anchor at renewal. An AI search API from a second vendor is cheap insurance against a single platform's outage or re-pricing.

Meter credits per job type. Instrument your own usage so you know what a search, a render and an extraction each cost you in production, not in the pricing table. That is the only way to notice when a bundle's convenient default quietly becomes your largest line item.

Treat the primitive boundaries as your portability seam. The old separations between search, crawl, extract and browser are still the natural places to swap a vendor. Keeping your own code organized around those boundaries, even while buying them from one platform, is what keeps the exit open. Our guide to build versus buy for web data and the browser infrastructure landscape both help you decide which layers to hold loosely.

What this means if you build on web data

• The unit of sale changed. You are buying a web data platform for AI agents now, not a search API or a scraper, so evaluate the whole bundle rather than one endpoint.
• Bundling is genuine leverage, not only lock-in. One bill, one auth and one fetch path is worth real money to a small team, especially as open-web access gets gated.
• Compare on jobs, not credits. A per-credit rate hides your true cost per answer, so normalize on your own workload before you commit.
• Keep an exit seam. Organize your code around the old primitive boundaries and keep a second provider warm, so the platform you chose stays a choice.