Web scraping tools with a Python SDK

Tavily

Real-time search API built for AI agents and RAG pipelines

Valyu

Search API spanning the web plus arXiv, PubMed, and SEC filings

Ceramic

Independent search index behind a single JSON endpoint for agents

Apify

Scraping platform with a marketplace of 10,000+ prebuilt Actors

Scrapling

Adaptive Python scraper whose selectors survive site redesigns

Context.dev

URL-to-markdown API with an added brand-data layer for agents

Browser Use

Hands any website to an AI agent, the top browser-automation repo

ScrapeGraphAI

Python library that scrapes sites from plain-English prompts

Scrapy

The original Python framework for large-scale web crawling

Playwright

Microsoft's cross-browser automation for testing and scraping

Crawlee

Apify's crawling library wrapping Playwright and Puppeteer

Selenium

Veteran browser automation with bindings for most languages

Crawl4AI

Open-source crawler shaped for RAG, the most-starred on GitHub

Beautiful Soup

Python parser that turns messy HTML into navigable trees

MechanicalSoup

Pairs Requests and Beautiful Soup for form-driven scraping

HTTPx + Parsel

Lightweight async scraping from HTTPx plus Scrapy Parsel

Trafilatura

Strips nav, ads, and chrome to leave the main article text