AI Crawler

An AI crawler is an automated client operated by a model provider or AI search company that fetches web pages for use in an AI system rather than for a conventional search index. The category matters to site owners because these bots now serve three purposes that used to be one: collecting text for model training, building a retrieval index that AI answers cite, and fetching a single URL because a specific user asked an assistant about it.

Separating the three is the whole exercise, because each carries a different bargain. A training crawler takes content into model weights, generally without attribution or referral traffic. A search crawler indexes pages so an assistant can cite and link them, which is the AI-era equivalent of being in the index at all. A user-triggered fetch is a proxy request on behalf of one person who is already looking at your link. Vendors now ship separate user agents for each role, so the three can be controlled independently in robots.txt. Blocking training while staying eligible for citations is a coherent policy; treating every AI bot as one bucket is the common configuration mistake.

Named agents, last reviewed July 2026. OpenAI runs GPTBot for training, OAI-SearchBot for the ChatGPT search index, and ChatGPT-User for user-triggered fetches. Anthropic runs ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for user-triggered fetches. Perplexity runs PerplexityBot for indexing and Perplexity-User for on-demand fetches. Google's regular Googlebot still does the crawling that feeds AI Overviews and AI Mode, while Google-Extended is not a crawler at all but an opt-out token controlling whether already-crawled content is used for Gemini training, and GoogleOther covers other internal uses. Apple runs Applebot for search and Applebot-Extended as a training opt-out token. Meta runs Meta-ExternalAgent for training and Meta-ExternalFetcher for user-triggered fetches. CCBot belongs to Common Crawl, whose archive functions as a training corpus for many labs rather than as a product of its own. Also in regular rotation: Amazonbot, DuckAssistBot, MistralAI-User, YouBot, and ByteDance's Bytespider, the agent most often named in publisher reports of robots.txt being ignored. Names and roles change; check each vendor's own documentation before writing a rule.

Compliance is voluntary and contested. robots.txt is a request, not an access control, and the record is uneven. Cloudflare published research in August 2025 alleging that Perplexity fetched pages through undeclared user agents after being blocked, a finding Perplexity disputed. Cloudflare also moved in July 2025 to block AI crawlers by default for new domains on its network and to pilot a pay-per-crawl mechanism, the first attempt at that scale to price the exchange rather than only permit or deny it.

For an AI product builder the relevance runs both ways. If you publish, robots.txt is now a distribution policy, and blocking the search-side agents removes you from AI answers as surely as a noindex tag removes you from Google. If you build, your own crawler should declare a real user agent, publish a page explaining what it does, honor robots.txt, and rate-limit itself. Vendors such as Firecrawl publish a named agent for exactly this reason. The alternative is being classified alongside Bytespider and blocked at the network edge before a request ever reaches an origin server.