One robots.txt line, four AI companies: what the shared AI training opt-out means for web data buyers

On 30 October 2026, Google, Microsoft and OpenAI each said they will honor a single robots.txt instruction for refusing AI training, and an Anthropic engineer told an IETF mailing list the same afternoon that his company had started building support for it. The robots.txt AI training opt-out is a group addressed to a policy token, __usage_ai-training, instead of to a named crawler. It is voluntary, it is still an Internet-Draft, and none of the four has said it is live.

If you buy web data rather than publish it, this is not mainly a publisher story. It is a new signal your vendor's fetchers will be expected to read, and a new way for the pages you depend on to say no.

What was announced

Google's Search Central post describes plans to support a shared AI training control in robots.txt, based on work from the IETF's AI Preferences working group. The text comes from draft-kelley-aipref-training-preference-signal, written by Kevin Kelley of Anthropic.

The mechanism reuses existing syntax. A site writes an ordinary group, but the User-Agent line names a use instead of a bot:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Path matching and longest-match precedence follow RFC 9309, the Robots Exclusion Protocol. The draft says it introduces no new directive names and no new grammar, so a parser that has never heard of the token skips the group like any other unfamiliar bot name. The double underscore is there because some deployed parsers match names loosely, and the prefix makes an accidental match unlikely.

PPC Land's detailed report has the rollout dates. Microsoft will honor the preference from 1 February 2027. Google says it aims to support the control in 2027, and its crawler documentation says early 2027. OpenAI says "the coming months". Anthropic says it has started implementing support, with no date.

Why a buyer should care

Until now, a publisher had to learn a list of crawler names (GPTBot, ClaudeBot, Google-Extended and so on) and add to it whenever a new one appeared. Any site that stopped updating its list left gaps, and vendors could point to them.

A single token makes the signal cheaper to send, so I expect more sites to send it. For a data buyer, that means the share of the web saying "not for training" will probably grow, in a uniform, machine-readable form. Two questions follow for any vendor in your stack:

• Does the vendor's fetcher parse the token at all, and what does it do when it finds it?
• Can the vendor tell you, per request, whether a page was fetched for a live answer, a cached index, or a training corpus?

The second question is the one we covered in Crawler purpose is now a product spec, where Cloudflare's defaults pushed vendors to declare a purpose. The robots.txt token is a separate route to the same distinction. A vendor running one undifferentiated fetcher fleet now faces two groups of publishers asking it to separate its traffic.

The definition matters. The draft borrows from the AIPREF vocabulary: AI training means using an asset to modify the learned parameters of a generative AI model. Using a page to answer a question at query time is not training under that definition, and neither is classification or ranking. A site can refuse training and still want to be fetched for retrieval, which is the case most search and extraction APIs are built around.

The wildcard rule is the catch

Most of the sites you scrape already have a robots.txt with User-agent: * and Disallow: / somewhere in it. The draft says wildcard groups must not be interpreted as matching the policy token. If no group names __usage_ai-training, no preference has been expressed.

So a site that blocks every bot today has, under this draft, said nothing about training. Kelley made the same point on the IETF list: a robots.txt file that doesn't use the mechanism, which describes every file published so far, expresses no preference. For a vendor, "we respect robots.txt" no longer settles the training question. It settles the crawling question, which is a different thing.

Older vendor tokens also do not carry over. A Google-Extended block keeps its meaning at Google, but it does not become a shared opt-out at OpenAI or Microsoft.

Google's mapping reaches into grounding

This is the part most likely to surprise an AI product team. Google says that when a site opts out of training with the shared token and has not mentioned Google-Extended explicitly, it plans to treat that as a Google-Extended disallow. Google-Extended is not only about training. Its crawler documentation ties it to grounding in Gemini Apps, and Google's post says sites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding there.

So a publisher who adds the shared token to block training may, at Google, also drop out of Gemini grounded answers unless it writes an explicit Google-Extended allow group. Google treats the explicit rule as the more specific one. AI Overviews and AI Mode in Search are governed separately, by Googlebot rules and a Search Console setting.

For a builder, the effect is indirect but real. If your product leans on Gemini grounding as a retrieval layer, the pool of citable pages could shrink in 2027 for reasons the publishers did not intend.

Microsoft is changing two things at once

Microsoft's Bing blog post, as PPC Land reports it, adds a page-level grounding preference, set by meta tag or HTTP header, separate from training. It also retires the NoArchive and NoCache tags it introduced in 2023 as AI controls. After 1 March 2027 it stops reading them as training or grounding signals, and sites that depend on them can record a temporary preference in Bing Webmaster Tools from 15 December.

Microsoft also says that once it detects either new preference on a site, it ignores any old tags still there. A publisher that adds only the shared training line could lose a grounding restriction it set earlier. If your pipeline uses Bing-backed search or grounding services, expect some pages to change status in March without any visible change on your side.

What it does not do

The limits are easy to gloss over.

• It is a request. OpenAI itself calls its participation a voluntary effort. Nothing in the draft forces any crawler to obey, and private licensing agreements can override it.
• It looks forward. Content collected before a site adds the token is already wherever it went.
• It has no formal standing. The datatracker lists it as an individual submission rather than a working-group document, and it expires on 3 May 2027 unless replaced.
• It is not universal. Meta has not said it will honor it, and its revised business terms for crawling sites tied to Facebook, Instagram and WhatsApp accounts now take effect on 20 November after publisher complaints. Amazon and Apple were not among the four.
• It does not replace existing controls. OpenAI says GPTBot rules stay in place, and Anthropic says ClaudeBot, Claude-User and Claude-SearchBot controls remain.

The token is still worth taking seriously. It sits under a larger compliance picture that already includes Cloudflare's purpose-based defaults and the provenance questions in our EU AI Act post. Google notes the plan fits the EU's code of practice for general-purpose AI models, which recognizes robots.txt as a primary way for publishers to state machine-readable preferences.

Questions to put to your vendor

Add these to the checklist in our web data vendor due diligence guide:

1. Parsing: will your fetchers read __usage_ai-training, and from what date? Microsoft's date is 1 February 2027. Ask for yours.
2. Wildcard behavior: what do you do with sites that block everything under User-agent: * but say nothing about training?
3. Purpose logging: can you record, per fetch, whether the content went to a live response, an index, or a training set?
4. Downstream use: if I pass your output to a model I fine-tune, whose preference binds the data, and where is that documented?
5. User agents: which user agents do you send, and are they separated by purpose?
6. Provenance: can you show me the robots.txt state in force when each page was retrieved? The draft says the applicable preference is the one in the file at retrieval time.

If you are building a corpus for model work, the web data for LLM training guide covers the sourcing side. I'd expect a corpus assembled after 2027 without any check of this signal to be harder to defend than one assembled before.

What to do this quarter

Nothing breaks today. Three things are worth doing before the 2027 dates.

First, find out which vendors in your stack fetch pages themselves and which resell someone else's index. The second group inherits the first group's compliance, and you will not see it unless you ask.

Second, check any workflow that depends on Gemini grounding or Bing-backed search for exposure to the March 2027 Microsoft change and the Google mapping.

Third, write down your own position. If you publish a site, you now have a quick way to state a training preference to four companies, and deciding it deliberately beats inheriting a default. The robots.txt glossary entry covers the syntax. Name the training group explicitly rather than relying on a wildcard.

Adoption is incomplete and the draft could still change. But four companies likely to touch your pipeline now agree on how a site says no to training, and that is new.