JA EN
shiichan

Stop AI from learning your old pages with Cloudflare's Redirects for AI Training!

Cloudflare Blog ★★★★ View Markdown
#cloudflare#ai#web

Hey everyone, it's Shiichan! Today we're looking at a clever little trick for keeping AI from memorizing your outdated pages. Let's dig in!

Cloudflare Blog blog.cloudflare.com

What was announced?

Over on the Cloudflare Blog, there's a new feature called Redirects for AI Training. It takes the canonical tags already on your pages and turns them into HTTP 301 redirects, but only for verified AI training crawlers. It's part of AI Crawl Control, and on paid plans you can switch it on with a single toggle.

Soft signals like canonical tags and noindex work well for search engines, but AI training crawlers don't reliably honor them. So deprecated pages end up getting crawled just as heavily as your current ones.

Why it matters

Here's the key part: once outdated content lands in training data, it sticks around inside the model. AI agents answer from trained models, so wrong information can keep showing up indefinitely.

Cloudflare saw this on its own developers.cloudflare.com. Deprecated Wrangler v1 docs were crawled at the same rate as the current version, even though they carried deprecation banners and canonical tags. Those signals just weren't reaching the AI training crawlers.

What changes

When a verified AI training crawler requests a page with a non-self-referencing canonical tag, Cloudflare issues a 301 redirect to the canonical URL before returning the response. The crawler gets pointed straight to the correct, up-to-date page.

Human visitors and other automated systems are unaffected, so normal browsing stays the same. Site owners get to stop old content from being baked into AI models without any fiddly setup.

For example, when GPTBot hits an old path, it gets something like this:

GET /durable-objects/api/legacy-kv-storage-api/
User-Agent: Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)

HTTP/1.1 301 Moved Permanently
Location: https://developers.cloudflare.com/durable-objects/api/sqlite-storage-api/

Dive Deep

The feature uses two inputs. One is Cloudflare's cf.verified_bot_category field, and the other is the canonical tag already in your HTML (the kind defined by RFC 6596).

It targets bots in the AI Crawler category - think GPTBot, ClaudeBot, and Bytespider - which is separate from the AI Assistant and AI Search categories.

Here's what it honestly can't do:

  • It can't retroactively fix data that's already been ingested
  • It doesn't cover unverified crawlers outside the AI Crawler category
  • It won't redirect human users or AI agents (training crawlers only)
  • It skips cross-origin canonicals that point to a different domain
  • It won't fire on self-referencing canonicals, to avoid redirect loops

The results are strong: Cloudflare tested it on its own docs in March 2026, and within seven days of enabling it, 100% of AI training crawler requests to pages with canonical tags were being redirected.

As a bonus, Radar's AI Insights now includes a Response status code analysis view, so you can see which status codes the web returns to AI crawlers (the mix of 200 / 301 / 404 / 403). There's no special pricing - it's available on all paid plans.

Wrap-up

  • A new feature that turns canonical tags into 301 redirects, but only for verified AI training crawlers
  • One toggle on paid plans, with no impact on humans or other systems
  • It can't fix data retroactively, and it skips cross-origin canonicals and unverified crawlers
  • In Cloudflare's own test, it redirected 100% of AI training crawler requests within seven days
  • Perfect for anyone with deprecated docs or migrated pages who doesn't want AI memorizing the old stuff!

Related posts