Free web scraping tools

SearXNG

Self-hosted metasearch that aggregates results from 70+ engines

Scrapling

Adaptive Python scraper whose selectors survive site redesigns

Common Crawl

Nonprofit web archive of over 10 petabytes behind most major LLMs

Marginalia Search

Search engine for the text-heavy old web Google buries

Stagehand

TypeScript and Python SDK for driving browsers with natural language

Scrapy

The original Python framework for large-scale web crawling

Playwright

Microsoft's cross-browser automation for testing and scraping

Puppeteer

Google's Node.js library for driving headless Chrome

Crawlee

Apify's crawling library wrapping Playwright and Puppeteer

Selenium

Veteran browser automation with bindings for most languages

Crawl4AI

Open-source crawler shaped for RAG, among the most-starred on GitHub

Beautiful Soup

Python parser that turns messy HTML into navigable trees

Cheerio

jQuery-style HTML parsing for Node.js, no browser required

Colly

Fast concurrent scraping framework for Go with a callback API

MechanicalSoup

Pairs Requests and Beautiful Soup for form-driven scraping

HTTPx + Parsel

Lightweight async scraping from HTTPx plus Scrapy Parsel

Trafilatura

Strips nav, ads, and chrome to leave the main article text

Mozilla Readability

The library behind Firefox Reader Mode, extracting article text

ClawBench

Benchmark testing browser agents on 153 tasks across 144 live sites

WebArena

812 long-horizon web tasks on reproducible self-hosted sites

WebVoyager

643 live-web tasks measuring end-to-end multimodal agents

Mind2Web

2,350 tasks across 137 sites measuring cross-domain transfer

OSWorld

369 computer-use tasks spanning Ubuntu, Windows, and macOS

Vals AI Web Search

Scores how web search tools lift agent accuracy on research tasks

SimpleQA

OpenAI benchmark of short factual questions, used to rank search APIs

Artificial Analysis Search Index

Same agent, swapped search API: quality, cost and speed