The web data tool index

Independent ratings for 72 search APIs, extraction tools, and browser infrastructure platforms. Updated daily.

Browse directory

RSS feed|Email digest|

github.com·ReleasesJust now

GitHub - browser-use/browser-use: 🌐 Make websites accessible for AI agents. Automate tasks online with ease.

github.com·ReleasesJust now

Hermes Agent Adds Scrapling Research Skill

skyvern.com·Research3h ago

Getting Claude to QA its own work

sciencedirect.com·Research1d ago

Systematic Review: Web Scraping Techniques, LLM Integration, and Legal-Ethical Considerations

lightpanda.io·Research1d ago

The Web Automation Stack Explained

skyvern.com·Releases2d ago

Skyvern Releases Updated Documentation

serpapi.com·Releases2d ago

Self-host SerpBear with Coolify

nimbleway.com·Research2d ago

Web Scraping GuideThe Web Scraper's Guide to Navigating Common PitfallsTom ShakedMarch 30, 20267min read

commoncrawl.org·Releases2d ago

Announcing a Change to Common Crawl Dataset Size Reporting

wpseoai.com·Legal2d ago

Legal Analysis: Is Scraping Google Search Results Permitted?

zyte.com·Research3d ago

AI is the new engine for web scraping

tavily.com·Releases3d ago

Tavily Launches CLI for Agent-Native Web Access and Search

View all 67 updates →Updated daily

How we evaluate

Same criteria, every tool. Each product is assessed on documentation quality, API design, reliability data, and pricing transparency. Affiliate relationships are disclosed and never influence placement or ratings.

For teams choosing infrastructure. Firecrawl or Tavily? Exa or Perplexity Sonar? We maintain assessments across the full landscape of tools connecting AI to the web, so you can evaluate options without running your own benchmarks.

Editor's picks

Featured tools

View all 72 tools →

Exa

Neural search engine using embeddings-based next-link prediction — finds semantically similar content, not just keyword matches.

FreemiumView details →

Tavily

Real-time search API purpose-built for AI agents and RAG pipelines, now owned by Nebius Group.

FreemiumView details →

Serper.dev

Fast, budget-friendly Google Search API at $1/1K queries — the go-to SERP API for AI startups and agent builders.

FreemiumView details →

DataForSEO

Comprehensive SEO data platform with SERP API plus 50+ additional APIs covering keywords, backlinks, and on-page data.

PaidView details →

Firecrawl

Converts websites to LLM-ready markdown via API, with crawling, extraction, search, and an agent endpoint — the Swiss Army knife of AI web data.

FreemiumView details →

Apify

Full-stack scraping platform with a marketplace of 10K+ pre-built scrapers (Actors), plus the open-source Crawlee framework.

FreemiumView details →

Directory

72 tools reviewed

AI-Native Search APIsFreemium

AI-Native Search APIsFreemium

Perplexity Sonar

AI-Native Search APIsPaid

AI-Native Search APIsPaid

AI-Native Search APIsPaid

AI-Native Search APIsFreemium

AI-Native Search APIsFreemium

AI-Native Search APIsFreemium

AI-Native Search APIsFreemium

AI-Native Search APIsFree

Writing

From the blog

Mar 15, 2026·6 min read

The AI search API landscape in 2026

A look at how AI-powered search APIs are reshaping the web scraping market, which players are leading, and what it means for teams evaluating infrastructure.

Mar 10, 2026·5 min read

Web scraping tools ranked by total funding

A breakdown of the most-funded web scraping companies, what they raised, and what the funding levels signal about the market.

Mar 5, 2026·6 min read

Comparing browser infrastructure for web scraping

Playwright vs Puppeteer vs Selenium vs cloud browser services — an objective comparison of browser automation options for scraping workloads.

Weekly briefing

Tool launches, funding rounds, pricing changes, and market data. Every Tuesday.