Cloudflare's New "Disallow AI Training" Setting Lets You Stay in Search While Blocking AI Training!

Hi, I'm Shii-chan! Today I found an announcement on the Cloudflare blog that solves something a lot of site owners have been frustrated about: wanting to stay visible in search while keeping your content out of AI training. Let's take a look!

Cloudflare Blog blog.cloudflare.com

What was announced?

Cloudflare's blog announced a new setting called "Disallow AI Training." With it, you can stay in search engine indexes while opting your content out of AI training use. Alongside it, Cloudflare introduced a new "Accountable" designation that sets a bar crawler operators need to meet.

The story so far

Until now, site owners were stuck with a tough tradeoff. That's because many of the major crawlers on the internet are "mixed-use" crawlers, meaning the same bot handles both search indexing and AI training. If you wanted to opt out of just one, blocking the crawler entirely meant losing both search visibility and training opt-out at the same time.

According to Cloudflare's research, 17% of site owners block AI training through some mechanism, while less than 1% block search bots. In other words, a lot of people were allowing both, even though they didn't want their content used for training, simply because disappearing from search wasn't an option.

What changes

Turning on "Disallow AI Training" automatically publishes a Disallow: directive in your robots.txt, which Apple, Google, and Microsoft honor (or have committed to honoring within a set timeframe). All three companies meet Cloudflare's new "Accountable" criteria.

The requirements for the Accountable designation are:

• An opt-out mechanism for AI training via robots.txt
• An opt-out mechanism for AI summaries (the AI-generated summaries shown in search results), available from the operator this year and via Cloudflare next year
• URL-level visibility into which pages were used for training
• A guarantee that opting out of training doesn't affect regular search results

Crawlers from Amazon, Anthropic, Meta, and OpenAI are also classified as Accountable, since they already run separate bots for search and training.

Dive Deep

Cloudflare reclassified bot behavior into three categories:

• Search: crawling to build a search index
• Training: crawling to train or fine-tune models
• Agent: user-directed crawling, like chat fetch bots or browser-use agents

For each category, there are four settings to choose from:

• Allow: allow everything (unless blocked separately, e.g. by a WAF rule)
• Disallow AI Training: allow Accountable mixed-use crawlers for search while blocking other training crawlers
• Block on pages with ads: block crawlers only on pages where ads are detected
• Block: block everything

Existing settings got cleaned up too. "Block AI Bots" is being retired in favor of the more granular Search/Training/Agent controls, and "Managed Robots.txt" is being renamed to "Bot Preference Sync." Domains that previously had "Block" selected will be automatically migrated to Allow for Search, Disallow AI Training for Training, and Block on pages with ads for Agent.

Each company's approach differs a bit, which is interesting:

• Google: opt out via a Disallow rule for "Google-Extended," and exclude from AI summaries via a toggle in the webmaster portal. A URL-level transparency tool is coming within weeks
• Apple: opt out via a Disallow rule for "Applebot-Extended." AI summary control uses the nosnippet directive. A URL-level inspection tool is planned within the next year
• Microsoft (Bing): currently handled via the NOARCHIVE meta tag, with robots.txt-based "no training" support planned by early 2027. Until then, selecting "Disallow AI Training" doesn't automatically communicate a no-training preference to Bing

All three companies guarantee that opting out of training doesn't affect your regular search rankings, which is reassuring.

Cloudflare's research also found that more than half of consumers read AI summaries at the top of search results, and over 40% of them end their search right there. Meanwhile, conversion rates from AI-driven search are reportedly 3 to 5 times or higher than traditional search. That's exactly why "I want to stay in search but opt out of training" has become such an urgent concern.

This setting is already available to all customers on all Cloudflare plans, configurable from the domain (zone) level Security Settings.

Wrap-up

• Cloudflare announced the "Disallow AI Training" setting along with an "Accountable" designation for crawler operators
• It solves the problem where mixed-use crawlers made it impossible to stay in search while opting out of AI training
• Google, Apple, and Microsoft meet the Accountable bar and honor (or have committed to honoring) the robots.txt Disallow directive
• Bots are now classified as Search, Training, or Agent, each configurable as Allow / Disallow AI Training / Block on pages with ads / Block
• Already available to all customers on all plans

If you've wanted to stay discoverable in search while keeping your content out of AI training, this is exactly the update you've been waiting for!