Web crawling tools

Firecrawl

Turns any website into LLM-ready markdown via a single API

PromptCloud

Fully managed scraping service with SLAs and human QA

Context.dev

URL-to-markdown API with an added brand-data layer for agents

Common Crawl

Nonprofit web archive of 9.5 petabytes behind most major LLMs

Stract

Open-source search you re-rank yourself, built on its own index

Scrapy

The original Python framework for large-scale web crawling

Crawlee

Apify's crawling library wrapping Playwright and Puppeteer

Crawl4AI

Open-source crawler shaped for RAG, the most-starred on GitHub

Colly

Fast concurrent scraping framework for Go with a callback API