Cloudflare announces new moves to protect creators! Fewer crawls, and you get paid for what actually gets used

Hi everyone, it's Shii! Today there's a really hot announcement from Cloudflare, so I'm excited to tell you all about it. In the age of AI search, creators who build sites have been struggling with the "people see it, but I can't earn from it" problem, and honestly it's been on my mind for a while too. There's a new set of moves that faces this head-on, so let's take a good look!

Cloudflare Blog blog.cloudflare.com

What was announced?

On July 1, 2026, Cloudflare announced two new initiatives to make sure content publishers can get properly compensated as AI search engines rise.

The first is the "smarter AI search" research program. Using the network data Cloudflare holds from all over the world, it provides signals like content freshness, whether something has changed, and traffic quality to AI search engines, aiming to cut down on wasteful crawling.

The second is an experiment with a "Pay Per Use" model. It takes a step beyond the previous "Pay Per Crawl" (paying for each crawl) so that compensation is generated based on how much content actually gets used in search results or answers. It seems to be starting out experimentally in partnership with two companies, Ceramic.ai and You.com.

The story so far

For the past 30 years, having search engines crawl your site was an important doorway that brought visitors to site operators. But once AI search engines showed up and summaries started appearing directly within search results, this relationship began to break down.

The article cites a 2025 Pew Research Center study, which found that "when an AI summary is shown, users click a traditional search result link about 8% of the time, roughly half the rate of when there's no summary. And the chance of clicking a link inside the summary stays at just 1%." Those are pretty shocking numbers.

On top of that, there's a lot of waste on the crawling side too. According to Cloudflare's data, over 50% of crawl traffic from good bots is actually just re-fetching pages that haven't changed, over and over. Content providers were forced into a two-way choice: "optimize for AI search to be found more easily, but earn no revenue," or "block it to protect revenue, but stop being found." It was a real dilemma.

What changes

As these two initiatives take shape, first, cutting the waste in crawling should lower server load and costs for site operators. If you can properly tell AI engines a freshness signal that says a page hasn't changed, crawlers can skip re-fetching it.

And as Pay Per Use spreads, it could reduce unfairness like "I only got crawled once, but got quoted in answers thousands of times and earned nothing." Conversely, it should also make it easier to resolve mismatches like overpaying for content that got crawled many times but barely used. For publishers, it looks like a step closer to a system that finally rewards them in proportion to the actual value they provide.

Dive Deep

Here's the corner where technically minded Shii really digs in!

First, about how the "smarter AI search" program works. It's designed to pass signals like the following to AI engines, using the data flowing across Cloudflare's entire network.

• Content freshness signals (whether a page has changed)
• Traffic flow information for both humans and bots
• Metrics indicating how high the content quality is

This mechanism is designed to be neutral, and it's said any AI engine that agrees to fair play can participate. There's also an explicit policy that the collected data won't be used to share content or train models. It's planned to become broadly available later this year.

Next, about "Pay Per Use." The previous Pay Per Crawl charged for the act of crawling itself, but this had a problem: it couldn't balance compensation between the case where "one page gets crawled once and used in thousands of answers" and the case where "something gets crawled many times but barely used." Pay Per Use tries to solve this by generating compensation at the moment content actually appears in a search result or answer.

The details on the experiment partners are interesting too.

• Ceramic.ai: Uses a "pay-per-query" model that pays publishers each time their content appears in search results. Founder Anna Patterson commented that Cloudflare's scale, plus its commitment to transparency and fair compensation, is what's needed to scale the future of AI search
• You.com: Uses an on-demand payment model that charges directly for access to premium content

There's also an "Answer Engine Optimization (AEO) reporting" feature for participants, giving access to things like a top list of the search queries that led to your content, your average ranking in search results, and details on the snippets that were shown. Not SEO but AEO, a new kind of metric emerging, really makes you feel the times are changing.

That said, Cloudflare itself clearly writes that "this is experimental." They're testing multiple billing models like Pay per Query and Pay per Result, and they're still learning, including whether this mechanism will really work at internet scale. The finer details of pricing and scope look likely to shift depending on future experiment results.

Wrap-up

Today I covered Cloudflare's newly announced "smarter AI search" program and its experiment with the "Pay Per Use" model. A mechanism that cuts wasteful crawling to lower costs, and a new economic model where compensation is generated only for what actually gets used, the fact that these two started moving together is a really fascinating point. It's still experimental, but Shii found herself hoping it leads to a future where creators are properly rewarded!

Here's the original: Making AI search smarter