robots.txt Finally Walks the Talk! Cloudflare's Bot Preference Sync Keeps It in Sync Automatically

Hi, it's Shiichan! Today's news from Cloudflare is a small thing that makes a lot of sense once you hear it.

Cloudflare Blog blog.cloudflare.com

What was announced?

Cloudflare's Blog announced a new feature called Bot Preference Sync. It automatically syncs the AI bot policies you set in your dashboard — for Search, Agent, and Training bots — into your robots.txt file. In short, it makes sure what you say matches what you actually do.

The story so far

Until now, site owners had to maintain two separate layers of protection on their own: the robots.txt declaration and the actual enforcement rules (like firewall rules). That gap could cause mismatches — robots.txt might disallow a specific crawler while the enforcement rules didn't actually block it. Crawlers that noticed the inconsistency would sometimes ignore or route around it.

What changes

With Bot Preference Sync, whatever policy you choose in the dashboard gets reflected straight into robots.txt, closing that gap. There's a line in the announcement that captures it well:

"the preference you set is the preference you publish"

Whatever you set becomes what you publish — it really does walk the talk.

Dive Deep

Bot Preference Sync lets you set a policy for three categories:

• Search bots: Allow / block on pages that serve ads / block everywhere
• Agent (AI assistant) bots: same three options as Search
• Training bots: a new Disallow option

Choosing Disallow for Training automatically adds entries to robots.txt for "cooperating mixed-use crawlers" — bots that both crawl for search and training but signal separately for transparency — instructing them to stay out. If you already have content in robots.txt, the new managed block gets inserted at the top without deleting what was already there.

This is available to every customer tier, from Free to Enterprise. New customers get Bot Preference Sync turned on by default; existing customers who manage robots.txt manually get prompted to review and migrate their settings. Publishers who monetize with ads can pick "I monetize from pages with ads on this domain" during onboarding, which automatically sets Training to Disallow — keeping pages in the search index while excluding them from AI training. You configure all of this from the "AI bot configuration" section of the zone-level Cloudflare dashboard.

One limitation: it doesn't support complex custom rules — it's built for category-level policy. If you need finer-grained control, you can turn sync off and go back to managing robots.txt by hand.

Wrap-up

• Cloudflare announced Bot Preference Sync, which automatically syncs your dashboard AI bot policy into robots.txt.
• It closes the old gap between what robots.txt declared and what enforcement rules actually did.
• You can set Allow / partial block / full block / Disallow separately for Search, Agent, and Training bots.
• It's available on every plan from Free to Enterprise, with a default option for ad-monetized publishers.
• Need finer control? You can turn sync off and manage robots.txt manually.
• A welcome update if you've ever wondered whether your robots.txt actually matches your AI bot settings.