Free web scraping tools

SearXNG

Self-hosted metasearch that aggregates results from 70+ engines

Scrapling

Adaptive Python scraper whose selectors survive site redesigns

Lightpanda

Zig-built headless browser using far less memory than Chrome

Common Crawl

Nonprofit web archive of 9.5 petabytes behind most major LLMs

Stract

Open-source search you re-rank yourself, built on its own index

Marginalia Search

Search engine for the text-heavy old web Google buries

Alexandria

Open, decentralized index aiming to rival Google and Bing

Gigablast

Long-running open-source search engine with its own web index

Stagehand

TypeScript SDK for driving browsers with natural-language steps

Scrapy

The original Python framework for large-scale web crawling

Playwright

Microsoft's cross-browser automation for testing and scraping

Puppeteer

Google's Node.js library for driving headless Chrome

Crawlee

Apify's crawling library wrapping Playwright and Puppeteer

Selenium

Veteran browser automation with bindings for most languages

Crawl4AI

Open-source crawler shaped for RAG, the most-starred on GitHub

Beautiful Soup

Python parser that turns messy HTML into navigable trees

Cheerio

jQuery-style HTML parsing for Node.js, no browser required

Colly

Fast concurrent scraping framework for Go with a callback API

MechanicalSoup

Pairs Requests and Beautiful Soup for form-driven scraping

HTTPx + Parsel

Lightweight async scraping from HTTPx plus Scrapy Parsel

Trafilatura

Strips nav, ads, and chrome to leave the main article text

Mozilla Readability

The library behind Firefox Reader Mode, extracting article text

ClawBench

Benchmark testing browser agents on 153 tasks across 144 live sites

WebArena

812 long-horizon web tasks on reproducible self-hosted sites

WebVoyager

643 live-web tasks measuring end-to-end multimodal agents

Mind2Web

2,350 tasks across 137 sites measuring cross-domain transfer

OSWorld

369 computer-use tasks spanning Ubuntu, Windows, and macOS