Pay per crawl is here: the web's new payment and verification layer for AI agents

A few months ago, in The web is closing as the agents arrive, we predicted that pay per crawl would move from experiment to normal. It has. What was a single Cloudflare product in mid-2025 is now a small stack of competing protocols for charging machines to read the web, and a second stack for proving which machines are allowed to. Bots crossed 57.4% of web requests in June 2026, the first time automated traffic outnumbered humans on the sites Cloudflare measures. That pressure is what produced the plumbing. For anyone building web access for AI agents, two new rails are worth understanding before your data costs or your access quietly break: a payment rail that decides what a fetch costs, and a verification rail that decides whether you get the fetch at all.

What is pay per crawl?

Pay per crawl is an arrangement where a website charges AI crawlers and agents for each request, instead of serving content for free or blocking it outright. The charge is usually signaled with HTTP 402 Payment Required. Five mechanisms now matter: self-hosted 402 gates, Cloudflare Pay Per Crawl, and the x402 standard handle payment; RSL states the license; PACTs handle verification.

The payment rail: three ways a page can charge your agent

The payment rail is the part most builders will hit first, because it is the part that returns an error your code has to handle. There are three live patterns, and they differ mostly in who sits in the middle.

Self-hosted HTTP 402 gates

The simplest version has no intermediary. The origin server returns 402 Payment Required with a machine-readable JSON body describing the price, a payment URL, and the terms. Your agent reads that, buys an access key (typically through something like Stripe), and retries the request with an Authorization bearer token. The server validates the key and returns 200 OK with the content.

This pattern is attractive to sites that want control and do not want to share a cut with a platform. For you, it means each gated source can have its own purchase flow and its own token to manage. There is no central directory, so discovery happens by hitting the wall and reading what comes back.

Cloudflare Pay Per Crawl

Cloudflare Pay Per Crawl, launched on July 1, 2025, moves the gate to the CDN. Site owners set a per-crawl price in their dashboard, connect Stripe, and let Cloudflare act as merchant of record and handle payouts. A crawler advertises its budget with a crawler-max-price header; if the price is within budget the request passes through, and if payment is required the edge returns HTTP 402. Cloudflare paired this with a default permission-based model for AI crawlers, meaning unrecognized bots are blocked unless a site opts them in.

Because Cloudflare sits in front of a large share of the web, this is the version most likely to show up across many sites at once with one billing relationship. That convenience is also the trade: you are transacting through one company's marketplace on its terms.

x402

x402 is the open-standard answer. Coinbase open-sourced it in May 2025, reviving the long-dormant 402 status code for internet-native payments, and governance has since moved to a neutral x402 Foundation. The flow mirrors the others: a server responds with a PAYMENT-REQUIRED header, the client signs a PAYMENT-SIGNATURE locally, and a facilitator settles the transaction. The difference is that settlement is on-chain, primarily in USDC on networks such as Base, Solana, and Ethereum. Coinbase also shipped a Payments MCP built on x402 in October 2025, wiring the scheme into the Model Context Protocol tooling that agents already use.

x402 is the option to watch if you want payment without depending on a single platform, at the cost of taking on a stablecoin and wallet dependency.

The licensing layer: RSL

Payment moves money. It does not, by itself, say what you are allowed to do with what you bought. That is what RSL, or Really Simple Licensing, is for. Launched on September 10, 2025 and managed by the nonprofit RSL Collective, co-founded by RSS co-creator Eckart Walther and former Ask.com CEO Doug Leeds, it provides machine-readable license terms that extend the robots.txt signal sites already publish. RSL supports a range of models: free, attribution, subscription, pay-per-crawl, and pay-per-inference, the last of which bills when a model uses the content to answer, not just when it fetches it. Publishers including Reddit, Yahoo, Quora, and Medium signed on early, with Arena Group, BuzzFeed, USA Today, and Vox Media following.

The distinction worth holding onto: RSL is the licensing layer, not the settlement layer. It declares the terms in a format an agent can parse, and it relies on something like the payment rail above to actually move the money. Treat the two as separate questions when you evaluate a source.

The verification rail: PACTs

The second rail is newer and points in a different direction. On June 22, 2026, Cloudflare announced Private Access Control Tokens, or PACTs, together with Google Chrome, Microsoft Edge, and Mozilla Firefox. The idea is attestation. A site with what Cloudflare calls "strong knowledge of personhood" issues an anonymous token, and browsers or designated bots present that token elsewhere so fewer identity checks are needed. Crucially, "personhood" is defined to include software authorized to act on behalf of a legitimate person, so a sanctioned agent can, in principle, carry a token too. Cloudflare CTO Dane Knecht framed it as removing friction for human and agent traffic alike without sacrificing privacy.

PACTs is not the same thing as anti-bot detection, which tries to guess whether a visitor is automated. It is closer to a passport: the site you arrive at trusts a credential issued earlier rather than re-testing you. The technical details are still being standardized through the Privacy Pass and anti-fraud working groups, so the shape can still change. The risk, as The Register noted, sits on the other side of the same coin: agents and behaviors that cannot get a token may simply be denied. A metered web that also requires a credential is one where unattested traffic degrades from "charged" to "refused."

What it actually costs

Here the honest answer is that nobody can hand you a tidy table. Per-crawl pricing is set by each publisher, and much of the infrastructure is still in beta or invite-only, so there is no reliable market rate to quote. Anyone publishing a single "cost per crawl" figure today is guessing.

What is clear is the shape of the change rather than the number. Web access is acquiring a per-request line item that sits on top of the compute you already pay for. For years the cost of web data was mostly labor, writing and maintaining scrapers. Now a metered access fee lands next to that, and it scales with how often your agents fetch. Agents fetch far more than humans do. That shifts the build-versus-buy calculation: the free side of "build" is getting a price tag, and a vendor that bundles access into a predictable subscription starts to look less like a markup and more like insurance.

How to architect for a metered web

You cannot control which protocol wins, but you can build so that any of them is survivable. Four moves:

Budget access as a recurring line item. Model per-request cost the way you model token cost, with a ceiling and an alert, not as a rounding error. An agent that fetches thousands of pages per task can run up a real bill.
Prefer tools and indexes that absorb the payment and attestation problem for you. A provider that owns an independent search index and sells you access has already solved the 402s and credentials on its side. You buy one contract instead of negotiating with every origin.
Instrument for 402s and denials. Your fetch layer should distinguish "blocked," "payment required," and "token rejected," and report them separately. Today these often collapse into a generic failure, which hides a cost decision you should be making on purpose.
Keep a fallback channel. Do not single-source anything that can start charging or refusing on short notice. Our guide to agentic web access covers the channel choices in more depth; the principle is to know your second option before the first one gates you.

What to watch

Two questions will decide how much of this you have to think about. First, whether the payment rail consolidates: three competing schemes is a tax on builders, and the market tends to pick one or two. Second, whether PACT-style attestation hardens from a friction-reducer into a gate, where an agent without the right credential is not just charged more but turned away. The web is starting to ask machines for both a ticket and an ID. The teams that handle the next year well will be the ones who designed for both before they were mandatory.