Content Signals Policy

The Content Signals Policy is Cloudflare's extension to robots.txt that lets a site owner state, in a machine-readable line, how a crawler may use content after it has been fetched. It defines three signals carrying yes or no values: search (build a search index and show links and short excerpts, explicitly excluding AI-generated summaries), ai-input (use the page as input to a live AI answer, which covers RAG and grounding), and ai-train (use the page to train or fine-tune a model). A publisher writes one line under a user-agent group, for example "Content-Signal: search=yes, ai-train=no, ai-input=no", and leaves any signal unset if it has no preference.

The policy exists because robots.txt was only ever able to express access, not use. A publisher could allow or disallow a fetch, but had no vocabulary for the distinction most publishers now care about: stay in Google's index, stay out of the training corpus. Cloudflare launched the format in September 2025 alongside its managed robots.txt feature, whose default block sets search=yes and ai-train=no while deliberately leaving ai-input unset, on the stated grounds that Cloudflare did not want to guess customer preference on that one. The signals are advisory. Nothing in the format blocks a request, and Cloudflare has been explicit that it is a reservation of rights rather than an enforcement mechanism. Whether that reservation carries legal weight has not been tested in court, and the format is a vendor extension, not a ratified addition to the Robots Exclusion Protocol.

For anyone building on web data, the practical consequence is that a crawl now has two separate compliance questions rather than one. Disallow rules govern whether you may fetch; content signals govern what you may do with what you fetched, and a page can legitimately be crawlable for search while flagged ai-train=no. If you are assembling a training corpus or a RAG index from your own crawler, or from a derived corpus such as Common Crawl, the honest posture is to parse the Content-Signal line and record it per document, not to treat a 200 response as blanket permission. Extraction APIs including Firecrawl and Crawl4AI respect robots.txt directives, but respecting a Disallow is not the same thing as honoring a use signal, so verify what your provider actually does.

Cloudflare announced further changes on 1 July 2026, reported by TechCrunch and others, that are scheduled to take effect on 15 September 2026. Per Cloudflare's own announcement, new domains, new sites added by existing customers, and existing free customers who have not changed their dashboard settings will default to allowing Search crawlers while blocking Training and Agent crawlers on pages that display ads, and mixed-use crawlers that give site owners no way to separate those functions will be blocked on ad-carrying pages. Cloudflare also added a "use" parameter (values immediate, reference, full) to the signal format. None of this has taken effect yet, and the classification details were still being finalized when the announcement went out, so treat the specifics as announced intent rather than shipped behavior.