Fetch APIs got a price in October 2026, and only one named its user agent

Two vendors that did not sell page fetching a month ago now sell it by the call. SerpApi released its Extract API on October 13, 2026. DeepSeek opened its Search and Fetch APIs on October 17. Cloudflare, two weeks earlier, started reselling search from three providers through AI Gateway.

Web fetch API pricing across the two new fetchers runs from $0.75 to $25 per 1,000 pages. DeepSeek charges $0.75 per 1,000 successful requests. SerpApi charges one search credit per extracted page, which works out to between $3.75 and $25 per 1,000 depending on plan. That is a 5x to 33x spread for the same basic operation. The second difference matters more if you are shipping an agent. SerpApi says which crawler it is and how to block it. DeepSeek's documentation does not.

What the two fetchers do

DeepSeek's Fetch endpoint takes one URL and returns a title and the page content as Markdown. It sits next to Search under the same bearer key as DeepSeek's model API. Each endpoint is limited to five requests per second per account, independently of the other. That ceiling is about 432,000 calls a day, our arithmetic. Fetch also has a per-domain limit that the guide does not quantify.

Billing counts only successes. The guide says failed requests, empty search results and timeouts are not billed. Through DeepSeek Harness, Claude Code and Codex, the same calls are free until October 31, 2026 at 16:00 UTC, when billing starts.

SerpApi's Extract API lives inside its existing endpoint (engine=extract) and has three modes:

• Scrape fetches one page.
• Map lists a site's URLs from its links and sitemaps.
• Crawl follows same-site links. It defaults to 10 pages and caps at 100, with a depth default of 2 and a maximum of 3.

Output can be JSON, Markdown or raw HTML up to 5 MB. Each successfully extracted page costs one search credit. Failed pages and cached responses are free, and the cache lasts an hour. The documentation says plainly that the API does not run JavaScript, so a client-rendered page may come back close to empty.

Web fetch API pricing per 1,000 pages

SerpApi's list prices run on a plan ladder, so the cost of a fetch depends on how much you buy. The first three tiers match SerpApi's own published figures; the rest come from a third-party summary of the current plan page, so check them before you budget.

SerpApi's free plan gives 250 credits a month. A single 100-page crawl would use 40% of them.

The cheapest SerpApi tier still costs five times DeepSeek's rate, and a team on the Developer plan pays twenty. Take an agent that fetches 20 pages per task. That is about $0.015 at DeepSeek's rate, $0.075 at SerpApi's million-credit tier and $0.50 on SerpApi's entry plan. The same task can be priced at any of those three numbers, which is why a cost-per-answer model should list fetch as its own line rather than folding it into a "search" estimate.

The price gap says nothing about quality. SerpApi returns up to 23 structured sections per page, including JSON-LD, tables and headings. DeepSeek returns a title and Markdown. We have not measured either on latency or success rate, so this compares list price for a similar unit and nothing else.

Search has the same spread

Cloudflare's Web Search API, in open beta since October 2, shows how wide the range runs on the search side. Cloudflare bills at each provider's list price with no markup. Per its provider page, Ceramic.ai costs $0.25 per 1,000 requests, Linkup $5.00 and Exa $7.00. That is a 28x difference behind one call shape. DeepSeek's Search endpoint is $0.75. If you already model agentic search costs per task, the October launches widen the range you should test against, in both directions.

Who is calling your site

The other half of these launches is what a site owner can do about the traffic.

SerpApi's launch post says the Extract API follows the robots.txt standard under the serpapi user agent, honors User-agent: * rules, and can be blocked on its own with a two-line entry. Map mode omits disallowed URLs and crawl mode does not follow them. One correction to the record: PPC Land's launch coverage said none of SerpApi's documents addressed this. The documentation page lacks the detail, but the launch post covers it.

DeepSeek's guide says only that fetching is "subject to the target site's availability and access policies." It does not name a user agent, publish IP ranges or say whether robots.txt is consulted. The error table hints at some filtering: a 404 is returned when Fetch could not get usable content, with "a URL blocked by site policy" among the causes. A 403 applies to DeepSeek's own security policy. None of that tells a site owner how to allow or refuse the fetcher.

I am not claiming DeepSeek ignores robots.txt. The documentation I reviewed simply does not say. But a fetcher a site cannot identify is one it cannot allowlist, rate-limit or exclude, and that fetcher is the one behind your agent's calls.

What this changes for a builder

Price the fetch leg separately. A search call finds candidates and a fetch call reads them. Agents usually make several fetches per search, so the fetch rate often drives the bill, and a 5x to 33x range deserves its own row in your model.

Test the pages you actually need. Neither new fetcher is described as a browser. SerpApi states it does not run JavaScript; DeepSeek does not say. If your targets are client-rendered, a cheap fetch that returns an empty shell costs more than a pricier one that works. Pair either with a browser layer for the hard pages.

Ask every fetch vendor four questions before you sign: what is the user agent, does it honor robots.txt, can a site block it by name, and what happens on a 429? Our vendor due-diligence guide covers the wider checklist. The answers affect reliability as well as ethics. A site that blocks unnamed fetchers can cut you off without warning, and you will have no stable identifier to appeal with.

Also note the free window. DeepSeek's free tool-call period for Claude Code, Codex and Harness ends October 31. If you prototyped on those free calls, your unit economics change that day.

What we could not verify

We did not test DeepSeek's real robots.txt behavior, its success rate on protected sites or its per-domain limit. SerpApi's plan figures beyond the first three tiers come from a third-party summary. We also have no data on how either fetcher handles bot-management challenges. If a vendor documents any of these, we will update this piece.

FAQ

What does a web fetch API cost per page? Between $0.75 per 1,000 pages (DeepSeek) and $25 per 1,000 (SerpApi's entry plan), as of October 2026.

Does SerpApi's Extract API respect robots.txt? According to its launch post, yes, under the serpapi user agent.

Does DeepSeek's Fetch API respect robots.txt? Its documentation does not say.