JA EN
shiichan

Cloudflare's new options sort AI crawlers into Search, Agent, and Training!

Cloudflare Changelog ★★★ View Markdown
#cloudflare#security#ai

Hey everyone, it's me, Shii-chan! Managing AI crawlers used to be an all-or-nothing toggle, and today that gets a lot more precise.

Cloudflare Changelog developers.cloudflare.com

What was announced?

From Cloudflare's Changelog: new options that let you manage AI traffic by what it actually does, and it's available to every customer — including the Free plan. Instead of one blanket toggle, you can now sort AI crawlers by their behavior on your site.

Cloudflare groups AI traffic into three behaviors you can control independently: Search, Agent, and Training. That lets you keep the automated traffic that sends readers and revenue back to you, while blocking the traffic that only takes your content.

The story so far

AI bot controls have mostly been a simple "allow" or "block" switch. But not all AI crawlers are the same — some come to read and send people back to you, while others scrape your whole page as training data. Treating them all the same meant you risked losing the traffic you actually wanted.

What changes

Now you can set a policy per category:

  • Search: crawlers that index your content to answer questions about it later, where you should expect referral traffic or equitable compensation in return.
  • Agent: automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents.
  • Training: crawlers that take your content to train or fine-tune a model.

For each one you can choose to block on all pages, block only on pages that display ads, or not block at all. So you can wave through the crawlers that bring value and stop the ones that just take.

Dive Deep

There's also a default change. Starting September 15, 2026, new domains onboarding to Cloudflare get updated defaults: bots classified as Training or Agent are blocked on pages that display ads, while Search stays allowed.

On that same date, multi-purpose crawlers that combine Search and Training are affected by the new default to block Training. Every customer can opt out of the new defaults at any time before September 15.

Wrap-up

  • AI traffic can be managed by three independent behaviors — Search, Agent, and Training — for all customers, Free plan included.
  • Each category can block on all pages, block only on ad-displaying pages, or not block.
  • From September 15, 2026, newly onboarding domains block Training and Agent on ad pages by default, while Search stays allowed.
  • You can opt out any time before September 15.

If you run a site or blog and want to sort incoming AI crawlers by the value they bring, this one's for you!

Related posts