Web Bot Auth: how AI agents prove who they are so the web stops blocking them

Web Bot Auth is a way for an automated client to sign every HTTP request with a cryptographic key so the site on the other end can confirm which operator sent it. Instead of trusting a user-agent string that anyone can type, a server checks a signature against a public key the operator has published. If you build an AI product whose agent fetches web pages, Web Bot Auth is the mechanism that lets your traffic be recognized and allowed rather than lumped in with the scrapers that sites are busy blocking. It is early and mostly vendor-driven, and it is worth understanding now.

The problem Web Bot Auth solves

For most of the web's history, a bot announced itself with a user-agent header. That string is a claim, not a proof. Anyone can send Googlebot or ChatGPT-User from any machine, and plenty of unwanted traffic does exactly that. The defensive answer has been IP allowlists: publish the address ranges your crawler uses, and ask sites to trust requests from those ranges. That works until addresses rotate, until an agent runs from a user's own device or a cloud function with a shared egress IP, or until a spoofer parks on a neighboring address in the same block.

This mattered less when bots were a minority of traffic. That is no longer the case. As we covered when bots crossed the halfway mark of all web requests, sites have started metering, authenticating, and walling off access by default. A blanket block is a blunt instrument, and it catches the agents you actually want to serve alongside the ones you don't. Both sides now need a way to tell a declared identity from a proven one. A guessable user-agent string cannot carry that weight.

What Web Bot Auth actually is

Web Bot Auth borrows a piece of standard HTTP plumbing: HTTP Message Signatures, defined in RFC 9421. The bot operator holds a private signing key and attaches a signature to each outbound request. The receiving server recovers the operator's public key, checks the signature, and now knows, with cryptographic certainty rather than a text-string guess, which operator sent the request.

The concrete pieces are small. The cryptography is EdDSA over Curve Ed25519. Three request headers do the work: Signature carries the signature itself, Signature-Input states what was signed, and Signature-Agent points to the operator's identity. The operator publishes its public keys as a JSON Web Key Set (JWKS) at /.well-known/http-message-signature-directory on a domain it controls. A verifier fetches that directory, matches the key, and validates the signature. There is no shared secret to leak and no address range to keep current, because possession of the private key is the whole proof.

The enrollment path

The operator side is a short sequence:

1. Generate an Ed25519 key pair.
2. Publish the public key as a JWKS at /.well-known/http-message-signature-directory on a domain you own.
3. Sign every outbound request, adding the three headers. Cloudflare ships Rust and TypeScript libraries so you are not implementing RFC 9421 by hand.
4. Submit your signed agent to a verifier's program so its network will recognize you.

On the verifier side, Cloudflare noted that applications arriving with well-formed Message Signatures are prioritized and approved faster, which is a gentle push to get the signing right before you apply.

Who is behind it, and who isn't

The timeline is short and almost entirely Cloudflare's. Cloudflare first floated Web Bot Auth in May 2025. On July 1, 2025 it folded HTTP Message Signatures into its Verified Bots Program, so a bot could enroll by signing rather than by registering IP ranges, starting on the Free and Pro plans with Business and Enterprise to follow. On August 28, 2025 it launched "signed agents" with a first set of partners: the ChatGPT agent, Goose from Block, Browserbase, and Anchor Browser, with Cloudflare's own Browser Rendering also appearing in the signed-agent category.

Then a second network joined. On November 21, 2025 AWS added Web Bot Auth support to AWS WAF, automatically allowing verified Web Bot Auth traffic by default for customers protecting Amazon CloudFront distributions, at no extra charge for the feature. Two large edge networks defaulting to "allow" for signed traffic is what gives the scheme its pull.

Here is the part the marketing write-ups get wrong. Web Bot Auth is not an IETF standard. The original architecture document, draft-meunier-web-bot-auth-architecture, expired and was replaced by draft-meunier-webbotauth-httpsig-protocol, whose latest revision is dated March 2, 2026. It is an individual submission. There is no chartered IETF working group behind it, and the draft itself carries the standard notice that it is not endorsed by the IETF and has no formal standing in the standards process. Several secondary articles describe a "chartered working group"; that is not what the datatracker shows. What exists is a de-facto convention that a few large companies have shipped in production, resting on a real ratified standard (RFC 9421) underneath. That distinction matters for planning: conventions can change faster than standards, and a design driven by two edge providers can shift with their commercial priorities.

What it changes if you build agents

If your product sends an agent out to fetch pages, signing that traffic moves you from the "declared" column to the "proven" one. You stop relying on a site guessing your intent from a header and start giving it a fact it can verify. On networks that default to allowing verified traffic, that is the difference between a served response and a challenge page.

This is the same shift we traced in the Amazon and Perplexity dispute, where the fight moved from access to identity once an agent was acting on a user's behalf. Web Bot Auth is the mechanical version of that shift for anonymous fetches. It does not grant permission on its own, and it is not a license to ignore a site's terms or its purpose-based crawler rules. What it gives a site is a reliable answer to "who is this," which is the precondition for any allow decision more precise than block-everything. A signature makes you legible to the site; it does not make you entitled to the page.

What it changes when your vendor adopts it

Most builders will meet Web Bot Auth through a vendor rather than by signing requests themselves. If you fetch through a managed browser or an extraction API, the signing key belongs to that vendor, and its identity is what the destination site sees. Browserbase and Anchor Browser were in Cloudflare's first signed-agent cohort, which is a concrete reason the choice of browser infrastructure for AI agents now carries an access dimension it did not have a year ago.

That turns a few questions into part of ordinary vendor due diligence:

• Do you sign outbound requests with Web Bot Auth, and are you enrolled in Cloudflare's Verified Bots program and recognized by AWS WAF?
• Is my traffic signed under your identity or can it carry mine?
• When a signed request is still blocked, what is the fallback, and do I see that in the response?
• How do you handle sites that treat signed agents differently by purpose or plan?

A vendor that has done this will answer quickly. A vendor that treats anti-bot detection purely as an evasion problem, rather than an identity one, is telling you where it will be when more networks default to allowing signed traffic and challenging the rest.

What to do now

You do not need to act on all of this at once. A short read of your own setup is enough. If you fetch first-party at meaningful volume, generating an Ed25519 key and enrolling is a low-cost hedge against the blocks that are spreading. If you fetch through a vendor, make Web Bot Auth support a due-diligence line item rather than signing anything yourself. And if your own fetches sit behind Cloudflare or AWS origins, know that both now treat signed traffic preferentially, which shapes what your users' agents will and won't reach.

The measured read

Web Bot Auth is a real answer to a real problem: the web needs to tell wanted automated traffic from unwanted, and user-agent strings cannot carry that load. Built on RFC 9421 and adopted in production by Cloudflare and AWS, it is credible enough to plan around. But it is a vendor-led convention, not a ratified standard, and the identity it proves is not the same as permission to proceed. Sign your traffic, or make sure your vendor does, and treat it as one signal among several rather than the settled shape of how agents will reach the web.