Web data tools for AI agents

Exa

Embeddings-based neural search that finds semantically related pages

Tavily

Real-time search API built for AI agents and RAG pipelines

You.com

Composable web, news, and research search APIs for enterprise agents

Parallel AI

Proprietary web index built for deep multi-hop agent research

Linkup

AI search that licenses publisher content instead of scraping it

Andi Search

Ad-free, privacy-first AI search that answers rather than lists links

Seltz

Web knowledge API running its own Rust crawler and hybrid index

Valyu

Search API spanning the web plus arXiv, PubMed, and SEC filings

Ceramic

Independent search index behind a single JSON endpoint for agents

NewsCatcher

Recall-first news and web search across 140,000+ sources

Serper.dev

Fast Google SERP API priced at a dollar per thousand queries

AnakinScraper

Extracts structured JSON with prebuilt, agent-ready site actions

Tabstack

Mozilla's web execution API routing from raw fetch to full browser

Context.dev

URL-to-markdown API with an added brand-data layer for agents

Browserbase

Cloud browser infrastructure for agents, with the Stagehand SDK

Steel.dev

Open-source browser API with stealth and session management

Anchor Browser

Browser automation for agents, and a verified Cloudflare bot

Hyperbrowser

Browser-as-a-service with natural-language control and fast launches

Airtop

Natural-language cloud browser with SOC2 and HIPAA compliance

Kernel

Autoscaling browsers with anti-bot detection and reusable sessions

Browser Use

Hands any website to an AI agent, the top browser-automation repo

Stagehand

TypeScript SDK for driving browsers with natural-language steps

Crawl4AI

Open-source crawler shaped for RAG, the most-starred on GitHub

ClawBench

Benchmark testing browser agents on 153 tasks across 144 live sites

WebArena

812 long-horizon web tasks on reproducible self-hosted sites

WebVoyager

643 live-web tasks measuring end-to-end multimodal agents

Mind2Web

2,350 tasks across 137 sites measuring cross-domain transfer

OSWorld

369 computer-use tasks spanning Ubuntu, Windows, and macOS

Vals AI Web Search

Scores how web search tools lift agent accuracy on finance and legal research