Embeddings-based neural search that finds semantically related pages
AI-Native Search APIs Real-time search API built for AI agents and RAG pipelines
AI-Native Search APIs LLM search API that returns cited answers instead of raw links
AI-Native Search APIs Composable web, news, and research search APIs for enterprise agents
AI-Native Search APIs Proprietary web index built for deep multi-hop agent research
AI-Native Search APIs AI search that licenses publisher content instead of scraping it
AI-Native Search APIs Reader API that turns any URL into clean markdown, now part of Elastic
AI-Native Search APIs Model-agnostic web search you add to any LLM in one call
AI-Native Search APIs Ad-free, privacy-first AI search that answers rather than lists links
AI-Native Search APIs Self-hosted metasearch that aggregates results from 70+ engines
AI-Native Search APIs Web knowledge API running its own Rust crawler and hybrid index
AI-Native Search APIs Search API spanning the web plus arXiv, PubMed, and SEC filings
AI-Native Search APIs Independent search index behind a single JSON endpoint for agents
AI-Native Search APIs Recall-first news and web search across 140,000+ sources
AI-Native Search APIs SERP scraping API covering 80+ search engines since 2017
Fast Google SERP API priced at a dollar per thousand queries
SEO data platform pairing SERP results with 50+ other APIs
Multi-engine SERP API with native LangChain and Haystack support
Low-cost SERP API with batch jobs up to 15,000 requests
SERP API wired into Zapier and Make.com for no-code pipelines
Volume-priced SERP API that drops toward 29 cents per thousand
Proxy aggregator and scraper monitoring with a SERP API attached
Google SERP API with image and reverse image search built in
Turns any website into LLM-ready markdown via a single API
Web Crawl & Data Extraction APIs Scraping platform with a marketplace of 10,000+ prebuilt Actors
Web Crawl & Data Extraction APIs Enterprise extraction platform from the team behind Scrapy
Web Crawl & Data Extraction APIs Handles proxy rotation and JS rendering behind one scraping API
Web Crawl & Data Extraction APIs Gets scrapers past CAPTCHAs, fingerprinting, and WAF blocks
Web Crawl & Data Extraction APIs Anti-detection scraping API with CAPTCHA solving at low cost
Web Crawl & Data Extraction APIs Fully managed scraping service with SLAs and human QA
Web Crawl & Data Extraction APIs Bundles anti-bot bypass, rendering, and screenshots into one API
Web Crawl & Data Extraction APIs Web data platform with native Databricks and Snowflake pipes
Web Crawl & Data Extraction APIs Adaptive Python scraper whose selectors survive site redesigns
Web Crawl & Data Extraction APIs Extracts structured JSON with prebuilt, agent-ready site actions
Web Crawl & Data Extraction APIs Rust page-to-markdown API that skips the headless browser
Web Crawl & Data Extraction APIs Mozilla's web execution API routing from raw fetch to full browser
Web Crawl & Data Extraction APIs URL-to-markdown API with an added brand-data layer for agents
Web Crawl & Data Extraction APIs Cloud browser infrastructure for agents, with the Stagehand SDK
Browser Infrastructure Open-source browser API with stealth and session management
Browser Infrastructure Headless browser platform for scraping, PDFs, and screenshots
Browser Infrastructure Browser automation for agents, and a verified Cloudflare bot
Browser Infrastructure Browser-as-a-service with natural-language control and fast launches
Browser Infrastructure Natural-language cloud browser with SOC2 and HIPAA compliance
Browser Infrastructure Zig-built headless browser using far less memory than Chrome
Browser Infrastructure Autoscaling browsers with anti-bot detection and reusable sessions
Browser Infrastructure Hands any website to an AI agent, the top browser-automation repo
Browser Infrastructure Anti-detect browser with fingerprint control and a headless API
Browser Infrastructure Patches Chromium to run automation that bot detectors miss
Browser Infrastructure API for the only independent Western search index at scale
Independent Web Indexes Nonprofit web archive of 9.5 petabytes behind most major LLMs
Independent Web Indexes UK search engine with its own index and a no-tracking pledge
Independent Web Indexes Structured feeds of news, forum, and dark-web content for enterprise
Independent Web Indexes Programmatic access to Yandex index, dominant across Russia
Independent Web Indexes Open-source search you re-rank yourself, built on its own index
Independent Web Indexes Search engine for the text-heavy old web Google buries
Independent Web Indexes French GDPR-native search running its own partial index
Independent Web Indexes Open, decentralized index aiming to rival Google and Bing
Independent Web Indexes Long-running open-source search engine with its own web index
Independent Web Indexes Vision and NLP parsing behind a 10-billion-entity knowledge graph
Agentic Extraction Python library that scrapes sites from plain-English prompts
Agentic Extraction TypeScript SDK for driving browsers with natural-language steps
Agentic Extraction Describe the data you need and get a REST API for any site
Agentic Extraction One API that generates self-updating extraction workflows
Agentic Extraction Automates multi-step data extraction across many websites
Agentic Extraction Auto-generates extraction logic with no selectors or code
Agentic Extraction AI agent that navigates and extracts using vision and LLMs
Agentic Extraction Query language for pulling structured data from any web page
Agentic Extraction Turns unstructured web content into structured enterprise datasets
Agentic Extraction Point-and-click robots that extract and monitor any site
Agentic Extraction The original Python framework for large-scale web crawling
Open Source Frameworks Microsoft's cross-browser automation for testing and scraping
Open Source Frameworks Google's Node.js library for driving headless Chrome
Open Source Frameworks Apify's crawling library wrapping Playwright and Puppeteer
Open Source Frameworks Veteran browser automation with bindings for most languages
Open Source Frameworks Open-source crawler shaped for RAG, the most-starred on GitHub
Open Source Frameworks Python parser that turns messy HTML into navigable trees
Open Source Frameworks jQuery-style HTML parsing for Node.js, no browser required
Open Source Frameworks Fast concurrent scraping framework for Go with a callback API
Open Source Frameworks Pairs Requests and Beautiful Soup for form-driven scraping
Open Source Frameworks Lightweight async scraping from HTTPx plus Scrapy Parsel
Open Source Frameworks Strips nav, ads, and chrome to leave the main article text
Open Source Frameworks The library behind Firefox Reader Mode, extracting article text
Open Source Frameworks Benchmark testing browser agents on 153 tasks across 144 live sites
812 long-horizon web tasks on reproducible self-hosted sites
643 live-web tasks measuring end-to-end multimodal agents
2,350 tasks across 137 sites measuring cross-domain transfer
369 computer-use tasks spanning Ubuntu, Windows, and macOS
