Open Source Frameworks

Crawl4AI

Open-source crawler shaped for RAG, among the most-starred on GitHub

Crawlee

Apify's crawling library wrapping Playwright and Puppeteer

Playwright

Microsoft's cross-browser automation for testing and scraping

Puppeteer

Google's Node.js library for driving headless Chrome

Scrapy

The original Python framework for large-scale web crawling

Beautiful Soup

Python parser that turns messy HTML into navigable trees

Cheerio

jQuery-style HTML parsing for Node.js, no browser required

Colly

Fast concurrent scraping framework for Go with a callback API

HTTPx + Parsel

Lightweight async scraping from HTTPx plus Scrapy Parsel

MechanicalSoup

Pairs Requests and Beautiful Soup for form-driven scraping

Mozilla Readability

The library behind Firefox Reader Mode, extracting article text

Selenium

Veteran browser automation with bindings for most languages

Trafilatura

Strips nav, ads, and chrome to leave the main article text