serp.fast

Long-running crawling API with markdown mode and an MCP server

Nathan Kessler
By Nathan KesslerUpdated

Each tool is evaluated against our methodology using public docs, vendor demos, and hands-on testing.

Some links on this page are affiliate links. We earn a commission if you sign up – at no additional cost to you. Our editorial assessment is independent and never paid. How we review.

Crawlbase website

What is Crawlbase?

Crawlbase, the company formerly called ProxyCrawl, sells a Crawling API that fetches a URL, optionally renders JavaScript, and returns HTML, JSON or GitHub-flavored markdown. It also publishes an MCP server with crawl, markdown and screenshot tools. Billing follows a graduated per-1,000 rate card and counts only successful requests.

Our verdict

Crawlbase has been running since 2017, and its billing is predictable: blocked requests are free and you can set a hard spend cap. The parts that matter for AI products are newer. format=md with a readability pass returns article text without the page chrome, and the crawlbase-mcp package gives Claude or Cursor a fetch tool with a few lines of config. The catch is cost on hard targets. Difficulty multipliers go up to 20x on the most defended sites, and JavaScript rendering doubles that, so price your real target list before you commit.

Categories:

Web scraping APIs abstract away the hardest parts of web data collection: JavaScript rendering, anti-bot detection, proxy rotation, and data parsing. Instead of building and maintaining your own scraping infrastructure, you send a URL and receive clean, structured data back. For AI applications, many of these APIs now return LLM-ready markdown or structured JSON.

Share:

Turns any website into LLM-ready markdown via a single API

Freemium155K+Web Crawl & Data Extraction APIs

Enterprise extraction platform from the team behind Scrapy

PaidMar 2026Web Crawl & Data Extraction APIs

Bundles anti-bot bypass, rendering, and screenshots into one API

FreemiumMar 2026Web Crawl & Data Extraction APIs

How Crawlbase compares

Firecrawl

Firecrawl is markdown-first by design and manages crawling and extraction for you, a closer fit if you want fewer moving parts.

Scrapfly

Scrapfly pairs a scraping API with its own anti-bot and proxy stack and gives detailed per-request cost reporting, useful when hard targets drive the bill.

Zyte

Zyte is the other long-standing vendor in this space, with automatic extraction for common page types on top of its fetch API.

Frequently asked questions

How much does Crawlbase cost?

Crawlbase uses one graduated rate card per 1,000 successful standard requests: $3 for the first 1,000, $2 for the next 10,000, $0.60 for the next 100,000 and $0.50 for the next million, falling to $0.02 at very high volume. Its own worked examples put 100,000 requests at $76.40 and a million at $527.50. Optional subscriptions run from $99 a month for 200,000 credits to $599 for two million. New accounts get 1,000 free credits, rising to 5,000 after setup steps.

Why do some pages cost more on Crawlbase?

Site difficulty multipliers apply automatically. Standard sites are 1x, and heavily defended ones step up through tiers to 20x at the top. JavaScript rendering then doubles whatever tier applies. Failed or blocked requests cost nothing, and the dashboard has an optional hard spend cap that stops usage before the bill passes it. Test a sample of your real targets first, because the multiplier mix decides your effective rate.

Can Crawlbase return markdown for an LLM?

Yes. Setting format=md on the Crawling API returns GitHub-flavored markdown. An extra readability flag removes navigation, sidebars, footers and ad slots before conversion, which suits articles and blog posts that are going into an LLM context. Response headers report the markdown features used and the resolved base URL, so you can turn relative links back into absolute ones.

Does Crawlbase have an MCP server?

Yes. The crawlbase-mcp package runs through npx and takes your normal and JavaScript tokens as environment variables. It exposes three tools. crawl returns rendered HTML, crawl_markdown returns cleaned markdown, and crawl_screenshot returns a full-page PNG the model can look at. It uses the same infrastructure as the Crawling API, so blocked requests are handled the same way.

Is Crawlbase the same company as ProxyCrawl?

Yes. ProxyCrawl renamed itself Crawlbase, and the markdown converter still reports itself as ProxyCrawl-API in a response header. Despite the old name, the core product is the Crawling API rather than raw proxies, which is why it's listed here. Crawlbase does also sell a residential Smart AI Proxy and managed scrapers for sites such as Amazon and Google.

Visit

Crawlbase

Visit