Crawler purpose is now a product spec: reading Cloudflare's September 15 default as a buyer

On 1 July 2026 Cloudflare put a date on something that had been drifting for a year: 15 September 2026. From that day, its defaults block AI training and agent use on pages that carry ads, and any crawler that cannot tell a site owner which of those things it is doing gets blocked outright on those pages. The press release is unusually specific about the category: "Mixed crawlers that do not give site owners the ability to choose between search, agent use, and training, will be blocked on all pages with ads."

The change applies to new Cloudflare customers, new sites created by existing customers, and all existing free customers. Settings stay editable in the dashboard, so this is a default rather than a ban, and a publisher who wants agent traffic can keep it. But defaults are what most of the web actually runs on.

For a buyer, the operative detail is not the deadline. It is that Cloudflare has turned purpose into something a vendor either declares or fails to declare, and has attached a penalty to failing. A web data provider that runs one undifferentiated fetcher fleet for search results, live agent fetches, and dataset construction now has an architectural problem with a due date on it. That is a procurement question, and September is close enough that you can ask it now and get a real answer.

The three-way split, and why "mixed" is the trap

Cloudflare's taxonomy has three buckets: search, agent use, and training. A publisher can now say yes to one and no to another. Wanting to appear in an AI answer engine's live results is a different decision from consenting to be in next year's pretraining corpus, and until recently there was no clean way to express the difference.

The mixed-use bucket is the dangerous one because it is not a purpose. It is a residual category for crawlers whose operator cannot or will not separate the traffic. If your vendor fetches a page once and uses that fetch for a real-time agent response, a cached search result, and a training set, it has no honest way to declare a single purpose, and under the September default it loses ad-supported pages entirely. Not throttled. Blocked.

The asymmetry matters more than it looks, because of where the volume sits. PPC Land, reporting Cloudflare's own figures alongside the July announcement, put training-related crawlers at 50.6% of the bot traffic Cloudflare classifies and search-related bots at 10.7%. Search, the purpose publishers are most willing to permit, is the smallest slice. Most of the crawl load comes from the purpose most likely to be refused.

Cloudflare's other number from the same announcement explains why publishers are receptive: "Over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages." Cloudflare says it is testing freshness and traffic signals with AI companies it did not name, to cut that waste. Read that as a hint about the next round of defaults. A crawler that re-fetches politely, respects crawl budget, and can prove what it did with the bytes will be treated differently from one that does not.

Whose web this actually removes

The deadline does not take the top 1,000 sites away from you. Large publishers were already negotiating individually, and a paying Cloudflare enterprise customer with tuned settings is not affected by a change to defaults.

What it removes is the long tail: free-plan sites, newly created sites, and anything a small operator spun up and never configured. That is the part of the web that nobody has a licensing deal with, and it is exactly the part that vertical coverage claims rest on. When a vendor tells you it covers regional job boards, niche forums, local news, or industry trade sites, it is describing the segment where this default lands hardest and where nobody will negotiate an exception on your behalf.

So a coverage number quoted in July may not survive September, and the loss will not be evenly distributed across your workload. It will concentrate in the sources you picked the vendor for.

Blocking was already asymmetric

None of this arrives into a neutral web. Access has been unevenly restricted for two years, and the unevenness runs along vertical lines.

TechnologyChecker's scan of 4,047 robots.txt files on 30 March 2026 found GPTBot named in 13.8% of them, ClaudeBot in 11.5%, and CCBot in 11.2%. Roughly one site in eight explicitly names the largest training crawler. The same report cites a separate BuzzStream analysis putting the figure at 79% for top news sites blocking at least one AI training bot. Different samples, different methods, not one census, so do not average them. But the gap between them is the finding: news publishers block hard, and the broad web mostly has not bothered.

The practical consequence is that any vendor's aggregate access statistic tells you very little. If you are building a news-grounded product, you are operating in the 79% world, and you should assume a vendor's headline coverage figure is carried by verticals you do not care about. If you are extracting from ecommerce or documentation sites, the picture is much better than the coverage discourse suggests. Read every claim against your own robots.txt reality, not the market average.

Web Bot Auth, in one section

Web Bot Auth is what turns a purpose declaration from a claim on a vendor's status page into something an edge network can check on every request.

A bot signs each request using HTTP Message Signatures (RFC 9421) with an Ed25519 key. It sends a Signature-Agent header containing an HTTPS URI that points to its own key directory, and it publishes a JSON Web Key Set at /.well-known/http-message-signatures-directory, itself signed. The receiving edge fetches the key, verifies the signature, and now knows with cryptographic certainty which operator sent the request. Cloudflare's implementation notes are exacting about the details, down to the header needing double quotes. Two IETF drafts cover the directory format and the request protocol; standardization is still in progress.

What this replaces is the user agent string, which was never evidence of anything. A user agent is a self-reported label that any client can set to any value, which is why "block GPTBot in robots.txt" was always a request rather than a control. A signature is not spoofable in the same way. That is the entire shift: identity moves from a claim to a proof, and once identity is provable, a purpose attached to that identity becomes enforceable.

Identity is not permission

Signing proves who you are. It does not entitle you to anything, and most coverage of Web Bot Auth runs the two together.

Cloudflare's own documentation frames Web Bot Auth as a way to verify that a request comes from a particular automated client, and treats access control as a separate layer. A perfectly signed, fully verified agent can still be blocked, rate limited, or charged, because the site owner's policy is evaluated after identity is established, not instead of it. Being identified gets a request considered; it does not get it served. We made a version of this argument in the Amazon and Perplexity dispute, where the fight was never really about whether the agent could be identified.

This matters for vendor evaluation because "verified" is becoming marketing vocabulary. The question that separates vendors is not whether they call themselves verified. It is whether they sign requests, whose key directory they sign with, and what purpose they declare when they do. A vendor that signs is legible to publishers and can be granted access deliberately. A vendor that does not sign is indistinguishable from unattributed traffic, and the trajectory of Cloudflare's defaults is not kind to unattributed traffic.

Adoption is uneven even inside single companies, which is the clearest evidence that signing is a per-crawler decision rather than a corporate posture. A claim circulating since May 2026 has Google's main indexing crawler not signing while its AI-browsing Google-Agent does. I could not trace it to Google's own documentation, so treat it as unverified rather than as a fact about Google. The structural point stands regardless: keys are published per crawler, so a company can sign one fleet and not another, and asking "does your company support Web Bot Auth" gets you a useless answer.

Six questions before September

Put these to any vendor whose output reaches production, in writing, and treat vagueness as the answer.

1. Which purpose do you declare, and can you declare more than one? Search, agent, and training are now separable. A vendor that cannot separate them is the mixed-use category by definition.
2. Do you sign requests with Web Bot Auth, and where is your key directory published? A URL is a verifiable answer. "We are a verified bot" is not.
3. Do you fetch on my user's behalf, or into a shared cache? A user-initiated fetch and a cache fill are different purposes with different permission profiles, and the answer also determines what happens to your data when another customer requests the same page.
4. Which CDNs and bot-management vendors list you, and under what identity? Cloudflare is the largest but not the only gate.
5. What is your coverage on ad-supported pages after 15 September, and how do you know? The honest answer involves a measurement plan, not a number quoted from a page written in June.
6. What happens to my SLA if a class of sites goes dark? Ask whether a blocked domain counts as a failed request, whether it is billed, and whether it appears in your error telemetry as something distinguishable from a timeout.

That last one is the cheapest to fix and the most commonly neglected. If your fetch layer collapses "blocked by policy" into a generic failure, you will discover this deadline as a slow quality regression rather than an event. The agentic web access guide covers the instrumentation in more detail.

Where the directory sits today

The shape of a good answer already exists in the market. Anchor Browser lists itself as a verified Cloudflare bot, and its CEO appears in Cloudflare's own signed-agents announcement, which is a specific, checkable claim rather than a posture. That is the bar to hold the rest to: Browserbase and Steel.dev in browser infrastructure, Firecrawl in extraction, Exa and Tavily in AI search, and the whole SERP data API group, where fetches are least distinguishable from one another and the mixed-use risk is highest.

What this means if you build on web data

Purpose separation is now a technical requirement, not a policy preference. A vendor whose architecture cannot distinguish a live agent fetch from a training fetch has an engineering problem due on 15 September, and you inherit its consequences.

Signing tells you who, not whether. Ask whether a vendor signs and what it declares. Verification status is a prerequisite for being granted access, never a substitute for being granted it.

Coverage claims expire on 15 September for a specific slice of the web. The exposure is the unconfigured long tail, which is where vertical coverage claims live. Re-verify the sources you actually depend on rather than the aggregate.

Instrument the difference between blocked, charged, and failed. This is the same recommendation we made about pay per crawl, and the reason has not changed: access decisions that arrive as generic errors become invisible, and the ones arriving this autumn are worth seeing.