Search API, crawl API or extract API: picking the right retrieval primitive

Four different things get sold as one category. A search API takes a query and returns ranked candidate URLs with snippets. A fetch or reader API takes one URL you already have and returns clean text. A crawl API takes a seed URL and walks a site. An extract API takes a page and returns JSON shaped like a schema you supplied. They bill in different units, fail in different ways, and belong at different points in an agent loop.

The question that separates them is whether you already know the URL. If you do not, you need search, and getting the actual page content is a second call. If you do know the URL and it is one page, you need fetch. If you know the URL but what you really have is a site rather than a page, you need crawl. Extract is the odd one out: it is not a way of getting bytes off the web at all. It sits on top of whichever of the other three produced the HTML and turns prose into fields.

Teams pick wrong in a predictable direction. They buy a crawler because the requirement was written as "we need web data", then use it to fetch single URLs at a crawl-shaped price. Or they buy a search API, ship it, and discover in production that a 200-character snippet does not contain the pricing table the user asked about. Both mistakes are cheap to avoid and expensive to run.

Four primitives, four jobs

The last column is the one vendor comparison tables leave out. Each primitive has its own characteristic silent failure, and three of the four return HTTP 200 while failing.

The question that decides it: do you know the URL yet

Run the decision in this order.

Do you know the URL? No means search. There is no way around it; an LLM guessing URLs from training data is a worse discovery mechanism than a $1-per-thousand SERP call. Yes means skip to the next question.

Is it one page or a site? One page means fetch. A site means crawl, and it means you need to decide the page budget before you start rather than after the invoice.

Do you need prose or fields? Prose means you are done: hand the markdown to your model. Fields mean an extract step, either a hosted extract endpoint or your own model call over the markdown you already paid for.

Does the page need a real browser? If the content only exists after JavaScript execution, or behind a login, the fetch layer is no longer a fetch layer. That is browser infrastructure, a different budget and a different latency class, covered in browser infrastructure for AI.

The reason to run it in that order is that each answer changes the unit you are billed in. Search is per query. Fetch is per page. Crawl is per page discovered, which is a number you do not control unless you cap it. Extract is per page multiplied by whatever the vendor charges for the inference step.

Search APIs return ranked candidates, not guaranteed full text

Two families sell under the search heading, and they hand back different things.

SERP APIs return what a search engine returned. Serper.dev is the volume end of this: its published pricing is $50 for 50,000 credits ($1 per 1,000) falling to $3,750 for 12.5M credits ($0.30 per 1,000), credits valid six months, as of July 2026. You get Google's ranking, organic results, and the answer-box and knowledge-panel fields when present. What you do not get is the page. SerpAPI vs Serper.dev covers the tradeoffs inside that category.

AI search APIs return results shaped for a model instead of a results page. Exa runs its own neural index and prices search at $7 per 1,000 with content at $1 per 1,000 pages, which is the clearest illustration of the split: retrieval and full text are separate line items. Tavily returns deduplicated, ranked content optimized for tokens rather than for a human scanning links. Brave Search API sits between the two, selling access to an independent Western index at a published $5 per 1,000 requests for Search and $4 per 1,000 plus token charges for its Answers product, with $5 in free credits applied monthly.

The practical consequence is the same across all of them. A search call gives you candidates and a snippet. If your agent needs to quote a specification, a price, or a date that lives in the middle of the page, the snippet will not contain it often enough to rely on, and you will make a second call to a fetch layer. Budget for two calls per useful answer, not one. For per-vendor latency, factuality and pricing detail, AI search APIs compared is the deeper page, and Exa vs Tavily is the head-to-head most buyers in this category actually run.

Reader and fetch APIs: one URL to clean markdown

Fetch is the cheapest and most commoditized primitive. You send a URL, you get markdown, you pay about a credit.

Jina AI made the interface famous by prefixing: put r.jina.ai/ in front of a URL and the Reader returns the main content as markdown. Billing is token-metered across Jina's whole Search Foundation API rather than per page, and every new key includes 10 million free tokens. Jina is now part of Elastic, which is worth knowing if vendor independence is a procurement criterion.

Webclaw takes the opposite architectural bet from most of the category: raw HTTP first, browser only as a fallback, with a Rust engine and an open-source core you can self-host. Its hosted plans start at a published $19 per month for 10,000 credits. Context.dev is a narrower product, URL-to-markdown with an added brand-data layer aimed at agents. Tabstack, built at Mozilla, routes each request from raw fetch up to a full browser depending on what the page needs, and advertises robots.txt compliance and ephemeral processing as defaults rather than options. In its Show HN thread a Tabstack team member described the billing as a simple credit model at $1 per 10,000 credits with 50,000 credits granted per account, while noting the long-term tiers were not finalized; treat that as an early-stage figure rather than a stable price.

Fetch APIs return HTTP 200 with the wrong content more often than they return an error, and that is the case to design around. Common cases: a JavaScript app shell where the content has not rendered, a cookie consent interstitial, a login wall, a bot-challenge page that renders as a short paragraph of text. All of these look like a successful extraction to a pipeline that only checks status codes. The cheap guard is a minimum content-length check plus a heuristic on the ratio of boilerplate to body, applied before the content ever reaches your model. Extracting clean markdown from any page goes into the output-quality side, and llm-ready markdown covers what "clean" is supposed to mean here.

Crawl APIs: traversal, budgets, and the pages you did not want

Crawl is the only primitive whose cost is unbounded by default. You give it a seed and rules; it decides how many pages that implies. On a documentation site that is a few hundred. On an e-commerce catalogue with faceted navigation it is effectively infinite.

Firecrawl is the reference implementation of a hosted crawl API and prices crawl at the same per-page rate as a single scrape, with a separate map endpoint that returns a site's URLs for one credit per call. Third-party reviews published in 2026 list its plans as $16 per month for 3,000 credits, $83 per month for 100,000, and $333 per month for 500,000 on annual billing; Skyvern's July 2026 review, written by a competitor, additionally reports that lower tiers cap crawls at 50 pages. Confirm current numbers on Firecrawl's own pricing page before you budget, because this is the most frequently recut pricing in the category.

Crawl4AI is the open-source alternative and changes the economics rather than the mechanics: you run it, so you pay in infrastructure and engineering time instead of per page, and the output is already shaped for RAG. Firecrawl vs Crawl4AI is the direct comparison. Apify approaches crawling from the marketplace side, with prebuilt Actors for specific sites that skip the traversal-design problem when someone has already solved your target. Scrapfly sits underneath all of this as the anti-bot and rendering layer for targets that block ordinary requests.

Three rules make crawl behave:

Cap the page count on every job. Not the depth, the absolute count. Depth limits are unintuitive on sites with wide navigation.

Map before you crawl. A map or sitemap call returns the URL list for roughly the price of one page. Filter it in your own code, then issue fetch calls against the survivors. This turns an open-ended job into a countable batch, and it is the single largest cost saving available in this primitive.

Respect the crawl budget you are consuming on the other side. Sites increasingly meter and block AI crawlers by identity rather than by rate, and a vendor that ignores robots.txt on your behalf is a liability you have taken on without pricing it.

Extract APIs: schema-shaped output and its silent errors

Extract turns a page into fields. You supply a JSON schema, the vendor runs a model over the fetched content, and you get back an object that validates. Firecrawl prices this at 5 credits per scrape against 1 for plain markdown, so extraction is roughly a 5x multiplier on the base fetch. Tabstack ships schema generation from a URL. Crawl4AI implements the same thing locally through its extraction strategies. The interface is uniform enough by now that switching vendors is a schema copy, not a rewrite.

The problem is not the interface, it is that schema validation proves shape rather than truth. A field can be populated, correctly typed, and wrong, and nothing in your pipeline will notice.

The size of that gap is measurable. Cleanlab's analysis of structured-output benchmarks reports per-field accuracy and whole-record accuracy separately, and the two diverge sharply: on their re-scoped insurance-claims extraction task, GPT-5 scored 0.956 field accuracy against 0.76 output accuracy, meaning roughly a quarter of records had at least one wrong field despite nearly every individual field being right. A 2026 arXiv benchmark on multi-source structured extraction makes the same point from the other direction, noting that existing benchmarks mostly test schema compliance and format adherence rather than whether the extracted values are correct.

For a buyer, that translates into two concrete design decisions. Monitor null rates per field over time, because the first symptom of a target page changing is a field that quietly starts coming back empty rather than an error. And decide which fields are load-bearing: an extracted product name that is wrong is a cosmetic bug, an extracted price that is wrong is a wrong answer shipped to a customer. Web extraction benchmarks covers how the public evaluations are constructed, and agentic web extraction covers the model-driven approach these endpoints wrap.

Composing primitives inside one agent loop

The standard production shape is search, filter, fetch, extract, and the filter step is the one teams skip.

Search returns ten candidates. Fetching all ten costs ten page credits and adds latency for nine pages the model will not use. Having the model rank the ten snippets first, then fetching two or three, is usually the difference between a workable per-query cost and one that does not survive contact with real traffic. The filter step costs one cheap model call and eliminates most of the fetch spend.

Crawl does not belong in a user-facing loop at all. It is a batch job that populates an index ahead of time. If an agent is triggering crawls in response to user input, the latency is unpredictable by construction and the cost is uncapped. Move it to a scheduled job and let the loop read from the index.

Cache aggressively at the fetch layer and key on the URL. Agents revisit the same handful of documentation pages, pricing pages, and API references constantly, and a fetch cache with a short TTL removes a large fraction of the calls without changing any answer.

Finally, keep the tool surface small. If you are exposing these primitives to a model through MCP, two well-described tools produce better routing than six overlapping ones. A model given both search and web_search_with_content will pick between them badly.

Cost and latency profile of each primitive

All figures are the vendors' own published rates or, where marked, third-party reports, and this category recuts pricing often. The general shape holds even when the numbers move: search and fetch are cheap and predictable, crawl is cheap per unit and dangerous in aggregate, extract is the multiplier.

Which vendors sell which primitives

A dash means the vendor does not sell that primitive as a documented endpoint, not that the result is impossible to approximate with the endpoints it does sell. Coverage reflects each vendor's own documentation as of July 2026. Crawl4AI is a library you run rather than an API you buy, which is why its cost profile is unlike everything else in the table.

The single-primitive vendors are the cheapest and fastest at their one job. The four-primitive vendors are the ones that cut integration work. Serper.dev at $0.30 per 1,000 at volume is not competing on breadth. Firecrawl at 1 credit per page across scrape, crawl and map is not competing on being the cheapest possible SERP call. Which of those matters depends on whether your bottleneck is unit cost or engineering time, which is the same question as build versus buy, one layer down.

The short version

Ask whether you know the URL. If you do not, buy search and accept that full content is a second call. If you do and it is one page, buy fetch and put a content-length guard in front of your model. If you do and it is a site, buy crawl but map first and cap the page count. Add extract only when you need fields, price it as a multiple of fetch, and monitor null rates per field because schema validation will not tell you when the values went wrong.

Then check whether one vendor covers your whole path. If your agent needs search plus fetch plus occasional extraction, a single-credit-pool vendor removes three integrations at some cost in best-of-breed quality. If retrieval quality is what makes your product work, keep search separate and commodity-shop everything below it. Both are defensible; the failure is buying a crawler and using it as a fetch API for eighteen months without noticing.