Notable repositories

Open-source async crawler optimized for LLM data extraction.

The fastest-growing AI scraping tool. Async-first, produces clean markdown for LLM pipelines. The open-source answer to Firecrawl.

LLM-orchestrated web scraping with automatic graph-based extraction.

Novel approach: uses LLMs to plan and execute scraping tasks. Impressive for complex extraction but LLM costs add up at scale.

AI framework that gives LLMs the ability to control a web browser.

The most-starred AI browser tool by a wide margin. Lets AI agents browse the web like a human. The future of web interaction, today.

Turn websites into LLM-ready markdown with a single API call.

The standard for AI-friendly web scraping. Open-source core, excellent hosted API. If you're building RAG or AI data pipelines, start here.

Natural language browser automation built on Playwright.

From the Browserbase team. Tell it what to do in plain English, it drives Playwright. Early-stage but the developer experience is unmatched.

Plugin framework for Playwright with stealth and ad-blocking plugins.

Stealth plugin makes Playwright invisible to most anti-bot systems. Essential for production scraping of protected sites.

curl with browser TLS fingerprints to bypass anti-bot detection.

Clever approach: make curl look like a real browser at the TLS level. Works surprisingly well against Cloudflare and Akamai.

Python web automation without a traditional webdriver dependency.

Successor to undetected-chromedriver. Removes the webdriver dependency entirely, making detection even harder.

Cross-browser automation library for Chromium, Firefox, and WebKit.

The backbone of modern scraping stacks. Microsoft-backed, fast, reliable. If you're doing JS-rendered scraping, you're probably using this.

Chrome DevTools Protocol automation for Node.js.

Still the most popular browser automation tool by stars. Playwright is technically superior but Puppeteer's ecosystem is massive.

Browser automation framework supporting multiple languages and browsers.

The grandfather of browser automation. Still relevant for legacy projects and teams with existing Selenium infrastructure. Modern projects should pick Playwright.

Custom Selenium chromedriver that avoids detection by anti-bot services.

Solves a real problem: getting past Cloudflare and similar anti-bot systems. Fragile by nature (Chrome updates break it regularly) but nothing else does this job.

Python library for pulling data out of HTML and XML files.

Every Python developer's first scraping tool. Simple, well-documented, battle-tested. For parsing, not crawling.

Fast, flexible jQuery-like HTML parser for Node.js.

The Node.js equivalent of Beautiful Soup. Incredibly fast for server-side HTML parsing. Pairs perfectly with Crawlee or raw HTTP requests.

High-performance XML and HTML processing library for Python.

When Beautiful Soup is too slow. C-backed, XPath support, handles malformed HTML. The performance choice for heavy parsing workloads.

Fast, high-level web crawling and scraping framework for Python.

The OG Python scraping framework. Mature ecosystem, steep learning curve, but nothing matches it for large-scale structured crawling pipelines.

Web scraping and browser automation library for Node.js.

From the Apify team. Best-in-class for JavaScript scraping with built-in Playwright and Cheerio support. The TypeScript-first approach is refreshing.

Elegant scraping framework for Go with a clean callback API.

If you're a Go shop, this is your only real option. Fast, concurrent, and well-maintained. Limited compared to Scrapy's ecosystem.

Python library for automating interaction with websites.

Simple alternative to Scrapy for small projects. Wraps requests + BeautifulSoup. Perfect for quick scripts, not for production pipelines.

Adaptive web scraping framework with smart element tracking, anti-bot bypass, and stealth browser mode.

Hit 31K stars within months of launch. The adaptive selector engine — finds elements even after a site redesign — is something no other framework does. Three fetcher modes, MCP server, BSD-3 licensed.