AI crawlers unmasked! Attribution Business Insights is here!

Hey everyone, it's Shii-chan! Have you ever wondered just how much AI crawlers are visiting your site? Today I found an announcement that turns that fuzzy feeling into a clear picture, so let me tell you about it!

Cloudflare Blog blog.cloudflare.com

What was announced?

On Cloudflare's official Blog, a new dashboard called Attribution Business Insights has arrived! It analyzes the bots (automated visitors) that come to your site and shows you which crawlers are helping your site and which ones are becoming a burden.

The announcement comes from Cloudflare's Blog, written by Jin-Hee Lee and Oliver Payne. It's not a short Changelog-style notice — it's a meaty post that explains exactly why this feature is needed.

Why it matters

Old-school search engines had a give-and-take relationship: they crawled your site, but sent people back to you through search results. Recent AI crawlers, however, keep pulling in your content while returning almost no traffic in exchange.

The crawl-to-referral ratios Cloudflare observed (how many crawls happen for each visitor sent back) ran from 118:1 all the way up to nearly 50,000:1! In other words, you could be crawled 50,000 times and get just one visitor back. That leaves site owners with a double burden: they lose referral visitors and pay more infrastructure cost to serve all that bot traffic.

What changes

Until now, it was hard to clearly see which bots were visiting, how much, and how much value they brought. With Attribution Business Insights, you can grasp that relationship at a glance!

And this visibility isn't just for looking at. When you talk with crawler operators about crawl compensation, you can bring concrete numbers — "my site is being crawled this much" — to the table. It means you can sit down at the business negotiation with data in hand!

Dive Deep

Here are the main things the dashboard shows you:

• Bot traffic volume — it distinguishes human visits from automated ones
• Crawl-to-referral ratios — calculated per bot operator across 24 hours, seven days, and 30 days
• Top bots breakdown — country of origin, bandwidth they consume on your site, and whether you're currently blocking or allowing them
• Classifier categories — crawlers are sorted into three buckets by purpose:
  • Training (crawling to train the next version of an LLM chatbot)
  • Search (refreshing databases for RAG)
  • Agent (agentic interaction that returns answers directly to an end user)

Technically, this feature works together with Cloudflare's existing Bot Management. That said, Attribution Business Insights itself is positioned as a hub for thoughtful, filtered analytics, and the actual blocking still happens over in Security rules — as the post puts it, "the action step still happens in Security rules." Splitting analysis from enforcement keeps things clean and easy to use.

It's available to all Cloudflare Bot Management customers, viewable from the analytics section of the Cloudflare dashboard. The post also references the Content Independence Day 2025 initiative around content flow.

Wrap-up

• Cloudflare announced the Attribution Business Insights dashboard — an analytics tool that visualizes AI crawler behavior and its impact on your site
• Observed crawl-to-referral ratios ran from 118:1 to nearly 50,000:1, showing AI crawlers tend to "take without giving back"
• You can see bot volume, crawl-to-referral ratios (24 hours / seven days / 30 days), top bots' country and bandwidth, and the Training / Search / Agent classification
• It works with Bot Management, splitting roles: analytics live in this dashboard, while blocking stays in Security rules
• It's available to Cloudflare Bot Management customers

This feature is a great fit for site owners curious about which bots are visiting and whether it's worth it, and for anyone who wants to negotiate crawl compensation!