Self-hosted web scraping tools

SearXNG

Self-hosted metasearch that aggregates results from 70+ engines

Firecrawl

Turns any website into LLM-ready markdown via a single API

Scrapling

Adaptive Python scraper whose selectors survive site redesigns

Webclaw

Rust page-to-markdown API with raw-HTTP-first browser-fallback engine

Steel.dev

Open-source browser API with stealth and session management

Browserless

Headless browser platform for scraping, PDFs, and screenshots

Lightpanda

Zig-built headless browser using far less memory than Chrome

Browser Use

Hands any website to an AI agent, the top browser-automation repo

Nstbrowser

Anti-detect browser with fingerprint control and a headless API

Common Crawl

Nonprofit web archive of over 10 petabytes behind most major LLMs

Marginalia Search

Search engine for the text-heavy old web Google buries

ScrapeGraphAI

Python library that scrapes sites from plain-English prompts

Stagehand

TypeScript and Python SDK for driving browsers with natural language

Kadoa

Enterprise web data extraction that auto-generates logic, no code

Skyvern

AI agent that navigates and extracts using vision and LLMs

Scrapy

The original Python framework for large-scale web crawling

Playwright

Microsoft's cross-browser automation for testing and scraping

Puppeteer

Google's Node.js library for driving headless Chrome

Crawlee

Apify's crawling library wrapping Playwright and Puppeteer

Selenium

Veteran browser automation with bindings for most languages

Crawl4AI

Open-source crawler shaped for RAG, among the most-starred on GitHub

Beautiful Soup

Python parser that turns messy HTML into navigable trees

Cheerio

jQuery-style HTML parsing for Node.js, no browser required

Colly

Fast concurrent scraping framework for Go with a callback API

MechanicalSoup

Pairs Requests and Beautiful Soup for form-driven scraping

HTTPx + Parsel

Lightweight async scraping from HTTPx plus Scrapy Parsel

Trafilatura

Strips nav, ads, and chrome to leave the main article text

Mozilla Readability

The library behind Firefox Reader Mode, extracting article text

ClawBench

Benchmark testing browser agents on 153 tasks across 144 live sites

WebArena

812 long-horizon web tasks on reproducible self-hosted sites

OSWorld

369 computer-use tasks spanning Ubuntu, Windows, and macOS