The directory for web scraping and AI data tools

Exa

Embeddings-based neural search that finds semantically related pages

Tavily

Real-time search API built for AI agents and RAG pipelines

Perplexity Sonar

LLM search API that returns cited answers instead of raw links

You.com

Composable web, news, and research search APIs for enterprise agents

Parallel AI

Proprietary web index built for deep multi-hop agent research

Linkup

AI search that licenses publisher content instead of scraping it

Jina AI

Reader API that turns any URL into clean markdown, now part of Elastic

LLMLayer

Model-agnostic web access you add to any LLM in one call

Andi Search

Ad-free, privacy-first AI search that answers rather than lists links

SearXNG

Self-hosted metasearch that aggregates results from 70+ engines

Seltz

Web knowledge API running its own Rust crawler and hybrid index

Valyu

Search API spanning the web plus arXiv, PubMed, and SEC filings

Ceramic

Independent search index behind a single JSON endpoint for agents

NewsCatcher

Recall-first news and web search across 140,000+ sources

SerpAPI

SERP scraping API covering 80+ search engines since 2017

Serper.dev

Fast Google SERP API priced at a dollar per thousand queries

DataForSEO

SEO data platform pairing SERP results with 50+ other APIs

SearchAPI.io

Multi-engine SERP API with native LangChain and Haystack support

ValueSERP

Low-cost SERP API with batch jobs up to 15,000 requests

HasData

SERP API wired into Zapier and Make.com for no-code pipelines

ScrapingDog

Volume-priced SERP API that drops toward 29 cents per thousand

ScrapeOps

Proxy aggregator and scraper monitoring with a free tutorial hub

Zenserp

Google SERP API with image and reverse image search built in

Firecrawl

Turns any website into LLM-ready markdown via a single API

Apify

Scraping platform with a marketplace of 50,000+ prebuilt Actors

Zyte

Enterprise extraction platform from the team behind Scrapy

ScraperAPI

Handles proxy rotation and JS rendering behind one scraping API

ZenRows

Gets scrapers past CAPTCHAs, fingerprinting, and WAF blocks

Scrapeless

Anti-detection scraping API with CAPTCHA solving at low cost

PromptCloud

Fully managed scraping service with SLAs and human QA

Scrapfly

Bundles anti-bot bypass, rendering, and screenshots into one API

Nimble

Web data platform with native Databricks and Snowflake pipes

Scrapling

Adaptive Python scraper whose selectors survive site redesigns

AnakinScraper

Extracts structured JSON with prebuilt, agent-ready site actions

Webclaw

Rust page-to-markdown API with raw-HTTP-first browser-fallback engine

Tabstack

Mozilla's web execution API routing from raw fetch to full browser

Context.dev

URL-to-markdown API with an added brand-data layer for agents

Browserbase

Cloud browser infrastructure for agents, with the Stagehand SDK

Steel.dev

Open-source browser API with stealth and session management

Browserless

Headless browser platform for scraping, PDFs, and screenshots

Anchor Browser

Browser automation for agents, and a verified Cloudflare bot

Hyperbrowser

Browser-as-a-service with natural-language control and fast launches

Airtop

Natural-language cloud browser with SOC2 and HIPAA compliance

Lightpanda

Zig-built headless browser using far less memory than Chrome

Kernel

Autoscaling browsers with anti-bot detection and reusable sessions

Browser Use

Hands any website to an AI agent, the top browser-automation repo

Nstbrowser

Anti-detect browser with fingerprint control and a headless API

Rebrowser

Patches Chromium to run automation that bot detectors miss

Brave Search API

API for the only independent Western search index at scale

Common Crawl

Nonprofit web archive of over 10 petabytes behind most major LLMs

Mojeek

UK search engine with its own index and a no-tracking pledge

Webz.io

Structured feeds of news, forum, and dark-web content for enterprise

Yandex Search API

Programmatic access to Yandex index, dominant across Russia

Marginalia Search

Search engine for the text-heavy old web Google buries

Qwant

French GDPR-native search running its own partial index

Diffbot

Vision and NLP parsing behind a 10-billion-entity knowledge graph

ScrapeGraphAI

Python library that scrapes sites from plain-English prompts

Stagehand

TypeScript and Python SDK for driving browsers with natural language

parse.bot

Describe the data you need and get a REST API for any site

Riveter

Prompt-driven web research that builds and enriches spreadsheet rows

Kadoa

Enterprise web data extraction that auto-generates logic, no code

Skyvern

AI agent that navigates and extracts using vision and LLMs

AgentQL

Query language for pulling structured data from any web page

Browse.ai

Point-and-click robots that extract and monitor any site

Scrapy

The original Python framework for large-scale web crawling

Playwright

Microsoft's cross-browser automation for testing and scraping

Puppeteer

Google's Node.js library for driving headless Chrome

Crawlee

Apify's crawling library wrapping Playwright and Puppeteer

Selenium

Veteran browser automation with bindings for most languages

Crawl4AI

Open-source crawler shaped for RAG, among the most-starred on GitHub

Beautiful Soup

Python parser that turns messy HTML into navigable trees

Cheerio

jQuery-style HTML parsing for Node.js, no browser required

Colly

Fast concurrent scraping framework for Go with a callback API

MechanicalSoup

Pairs Requests and Beautiful Soup for form-driven scraping

HTTPx + Parsel

Lightweight async scraping from HTTPx plus Scrapy Parsel

Trafilatura

Strips nav, ads, and chrome to leave the main article text

Mozilla Readability

The library behind Firefox Reader Mode, extracting article text

ClawBench

Benchmark testing browser agents on 153 tasks across 144 live sites

WebArena

812 long-horizon web tasks on reproducible self-hosted sites

WebVoyager

643 live-web tasks measuring end-to-end multimodal agents

Mind2Web

2,350 tasks across 137 sites measuring cross-domain transfer

OSWorld

369 computer-use tasks spanning Ubuntu, Windows, and macOS

Vals AI Web Search

Scores how web search tools lift agent accuracy on research tasks