Pay Per Crawl levels up! Now you can choose free or paid, page by page

Hi, I'm Shiichan! Today I found a fresh update to Cloudflare's Pay Per Crawl, so let me tell you about it!

Cloudflare Changelog developers.cloudflare.com

What was announced?

The Cloudflare Changelog announced advanced configuration options for Pay Per Crawl, a feature of AI Crawl Control. Pay Per Crawl is the mechanism that lets you charge AI crawlers for accessing your site's content.

With this update, you get finer control at the page and request level, on top of the simple zone-wide setting that existed before!

The story so far

Until now, Pay Per Crawl mostly worked by applying a single pricing rule to an entire zone. That made it hard to say "this page should stay free, but that one should be paid." Pricing was largely fixed too, with no way to vary it dynamically based on the request.

What changes

This update brings two big capabilities.

• You can disable Pay Per Crawl for specific URI patterns using Configuration Rules, so you can leave "discovery" pages like your homepage or category listings free while still charging for the actual content pages.
• Your origin server can return a crawler-price response header, or you can use a Cloudflare Worker, to set pricing dynamically per request.

In short, site owners can now design exactly which parts of the site are free and which parts earn money.

Dive Deep

Here's how the dynamic pricing works.

• Cloudflare attaches a cf-pay-per-crawl header to requests sent to your origin. Its value is one of pricing=zone-default, pricing=in-band, or pricing=bypass, so your origin or Worker can tell which pricing mode is currently active.
• If your origin returns a header like crawler-price: USD 3.14 in its response, that price is used for billing the crawler.
• With a Worker, you can also branch pricing by request path — for example, charging for /premium-content/ while keeping /free-content/ free.

Disabling Pay Per Crawl for a pattern is as simple as picking "Disable Pay Per Crawl" from a Configuration Rule in the dashboard. Operationally essential paths like robots.txt, sitemap.xml, and /.well-known/security.txt are already excluded from billing by default.

One thing to note: Pay Per Crawl itself is still in private beta, so you'll need to sign up to use it.

Wrap-up

• Pay Per Crawl now supports free access by URI pattern
• Dynamic per-request pricing is possible via an origin header or a Worker
• Cloudflare tells your origin the active pricing mode through the cf-pay-per-crawl header
• Still private beta, but a must-check update if you're thinking about AI crawler control or monetization