You can now sort AI bots into Search, Agent, and Training! And ad pages get protected by default

Hey there, it's Shii! Today there's some really exciting news from Cloudflare, and I just can't hold back my excitement. The way we deal with AI bots has gone from "block or allow, take your pick" to something you can tune in much finer detail. Let me tell you all about it!

Cloudflare Blog blog.cloudflare.com

What was announced?

On July 1, 2026, timed to its second "Content Independence Day," Cloudflare announced a new feature that lets website operators manage AI traffic in much finer detail. It's been a year since the original "Content Independence Day" declaration, so this feels like the evolved version finally making its debut.

The key point this time is that you can now split AI bot access into three purposes, "Search" (collecting content to show in search results), "Agent" (an agent doing something in real time on behalf of a human), and "Training" (collecting data to train or fine-tune a model), and choose to allow or deny each one individually. And a big point is that features which used to be limited to certain plans are now open to all customers.

Alongside this, Cloudflare also announced "BotBase," which gives visibility into bots for Enterprise Bot Management customers, plus a mechanism that extends robots.txt so you can even specify how your content gets used. That's a lot to unpack!

The story so far

The way it used to work, crawlers would gather content and, in return, send referral traffic back to the site, a give-and-take relationship that held up for decades. But today's AI crawlers increasingly just soak up content without giving anything back, and that's left operators of smaller sites in a really tough spot.

To make things worse, there was also this dilemma: "if you want to show up in search results, you have to accept being used for AI training too." There were ways to block bots, but usually it was only "block everything all at once," which risked losing your discoverability through Search along with it. Not being able to choose by purpose was pretty inconvenient.

What changes

From now on, site operators can set things up by purpose in fine detail, like "allow Search but decline Training." You can keep the traffic coming from search engines while blocking only the part that would use your content for model training without asking. That kind of choice is now possible.

What's especially worth watching is the new default setting that takes effect on September 15, 2026. For pages monetized with ads, new domains will block Training bots and Agent bots by default, while continuing to allow Search bots. The reasoning goes like this: "ads are meant to be seen by humans. So keep Training and Agent bots, which could steal away human attention, at a distance." Existing customers can change the setting anytime before September 15, and choosing to do nothing is also an option.

Dive Deep

This is the part where technically minded Shii really gets going!

First, to recap the three categories: Search is "scanning and indexing a site to display it in search results," Agent is "automated action carried out in real time on behalf of a user (chat fetch bots like ChatGPT, or agents like Gemini and Claude operating a browser)," and Training is "a crawler fetching content to train or fine-tune a model." Training is distinctive in that the data gets permanently absorbed into the model.

Next, a new parameter called "content use" also appeared. For bot management customers, you can now specify the level of content usage in three tiers.

• immediate: interact on the spot only, with no storage or reuse
• reference (default): indexing, excerpting, and linking back are all OK
• full: summarizing and reproducing are OK

When you build this into "Content Signals," the extended robots.txt spec, you can write it like this.

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

The point is the newly added use field. That said, just like the other directives in robots.txt, keep in mind this is only an "expression of preference," not an enforced block.

Even more interesting is the idea of "transitive trust." Using the Forwarded header from RFC 7239, you can flexibly manage access even when it comes through an intermediary.

Forwarded: for="openai";use="reference"

Writing it like this lets you manage trust in stages, something like "I trust OpenAI, but that doesn't mean I unconditionally trust every developer who uses OpenAI's tools." It's a really thoughtful design.

The meaning of Verified bots changes too. Until now, being Verified basically meant being allowed by default, but from now on Verified means "may be allowed," and whether it's actually allowed is decided by the category-side settings, like whether you allow Search. To become Verified, there seem to be two conditions: honestly showing who you are, and not abusing the access you gained through that honesty.

For bots that serve multiple purposes at once, like Googlebot, Applebot, or Bingbot, there's a principle that "the most restrictive rule applies." For example, if your setting blocks Training, a bot that does both Search and Training gets blocked. It's a bit strict, but it makes sense to err on the side of caution.

As for how to configure it, existing customers can make changes by opening "Manage AI traffic" in the zone Settings on the Cloudflare dashboard. New users will automatically get the new defaults from September 15, 2026 onward. Enterprise customers can use a dashboard called "BotBase" to browse a searchable catalog of known bots, filter traffic by category, and copy Detection IDs to use in security rules. As for pricing, there's no mention of extra charges, and these management options are said to be available across all customer tiers, including the free plan.

Wrap-up

What I covered today was Cloudflare's new option to sort AI bot access into three types, Search, Agent, and Training, and manage each individually. A big point was that on ad pages, Training and Agent will be blocked by default starting September 15, 2026. There's even the robots.txt content use signal and the transitive-trust mechanism using the Forwarded header, so it's not just about blocking, it opens up finer choices, which is something Shii is really happy about. For site operators, being able to choose for yourself how you deal with AI feels really reassuring.