Web Bot Auth

Web Bot Auth is a scheme for cryptographic bot identity: an automated client signs its outbound HTTP requests with a private key, and the server it is fetching from verifies that signature against a public key the operator has published, establishing who is actually making the request. It is built on HTTP Message Signatures (RFC 9421), uses an Ed25519 keypair in the common profile, carries a Signature-Agent header pointing at the directory where the operator's keys live, and expects that directory to be served as a JWKS document.

The shift it represents is from blocking by inference to allowing by identity. The signals sites have used to identify crawlers are the User-Agent string, which is a self-declaration any client can forge, and the source IP address, which is verifiable only where an operator publishes its ranges and supports reverse DNS confirmation, as Google and Bing do for their crawlers. Neither scales to a web where a large and growing share of traffic is agents fetching on behalf of individual users. A signature moves the question from "does this request look like a crawler" to "which operator's key signed this", which is answerable. That in turn makes finer-grained policy possible: a publisher can allow a verified archiver, rate-limit a verified AI crawler, block an unverified one, and price access per identity instead of applying one blanket rule to all non-browser traffic.

Standardization is live but not finished, and the details are still moving. The IETF has chartered a Web Bot Authentication working group (webbotauth) in the Web and Internet Transport area, chaired by David Schinazi and Rifaat Shekh-Yusef under area director Mike Bishop, with an approved charter to standardize cryptographic authentication of automated clients and the conveyance of information about their operators. The charter names search crawlers, web archivers, AI training crawlers, and link checkers as in scope, and explicitly excludes end-user authentication and non-human-facing API auth. It also lists coordination with the AIPREF, HTTPBIS, OAUTH, TLS, and WIMSE groups, which is a fair map of where the adjacent decisions get made. The earlier architecture draft from Thibault Meunier and Sandor Major expired and was superseded by a working-group protocol document, so treat specific header names and profile details as provisional until the working group's deliverables land.

For anyone building an AI product that fetches, identity cuts both ways. Publishing a key and signing your requests is how you stop being indistinguishable from anonymous scrapers and become an entity a publisher can allow, meter, or bill on its own terms. It is also how you become attributable, which means the same mechanism that gets you through the door lets a site throttle you specifically, charge you specifically, or revoke you specifically. Infrastructure is already moving this way at the vendor layer: Anchor Browser operates as a verified Cloudflare bot, and Common Crawl has fetched under one published, documented crawler identity since 2008, which is roughly what a signed identity formalizes.