One year since Cloudflare's "Content Independence Day"! I looked at how the web economy started moving in the age of AI agents
Hey there, it's Shii! Today I read the one-year anniversary report on Cloudflare's "Content Independence Day," and the numbers were so staggering that I couldn't stop feeling excited. Let's take a peek together at how money flows in an era where AI agents roam the web!
Cloudflare Blog
blog.cloudflare.com
What was announced?
Cloudflare declared "Content Independence Day" in 2025, and has been building a mechanism that lets you block AI training crawlers by default for new domains. This time, as a one-year retrospective report, they've pulled together how much the market for monetizing content has actually grown.
According to the report, regular users of generative AI have already surpassed 2.5 billion, which is 30% of the world's population. And this adoption pace is more than twice as fast as smartphone adoption, reaching this point in just 3.5 years. That's an incredible sense of speed, isn't it?
Even more striking is that non-human traffic on the internet, meaning access by AI agents and crawlers, has exceeded 50% of the total for the first time. The way people search for information itself is changing, to the point that "of every hour spent looking something up online, only 15 minutes goes to the open web."
The story so far
Until now, it was normal for AI companies' crawlers to crawl the entire web for training with no compensation to the publishers and site operators who created the content. From the site's side, you couldn't see how your content was being used, and you got barely any payment for it, a frustrating situation that dragged on.
On top of that, there was the problem that it was hard to tell whether a crawler was "for search," "for AI training," or "for generating an agent's answers." In particular, "mixed-use crawlers" that blend search and AI experiences gave site operators no way to distinguish intent, so they couldn't even be used as leverage in negotiations.
What changes
Over this past year, a trend has actually started to form where increasing transparency and making content scarcity visible translates into bargaining power for publishers. Since 2023, more than 50 licensing agreements have already been signed between publishers and AI companies. That means a mechanism where creators actually get paid is gradually taking shape.
On the other hand, some sites in the most-crawled categories saw human visits drop by as much as 40% in less than a year. Even though licensing revenue has emerged, it doesn't look like it can straightforwardly replace the referral traffic and ad revenue of the past, so for site operators it still feels very much like a transitional period.
Dive Deep
From here, let me dig properly into the more technical side.
First, about crawler classification. On Cloudflare's network, AI training crawlers made up 52% of all crawler requests (as of June 2026), more than double the 22% figure from spring 2025. Another key point is that "mixed-use crawlers," used for both search and AI experiences, account for more than 36% of the total. Google's crawler is the representative example of this mixed-use type, holding about 88% of all referral traffic while reportedly accessing about twice as much information as other AI companies. Because site operators can't tell whether this crawler is for search or for AI training, it poses a major challenge on the transparency front. Cloudflare has set a goal of bringing the share of these mixed-use crawlers close to zero over the next year.
Next, the new infrastructure and products that have emerged. One is Monetization Gateway, a mechanism that uses an open protocol called x402 to settle payments in a stablecoin for web pages, datasets, APIs, and MCP tools sitting behind Cloudflare. Another is Attribution Business Insights, a dashboard-like feature that makes visible with what intent and how much crawlers are accessing, and the value of that access. By combining these with the existing Pay per Crawl (a billing model based on the number of crawls), the aim is to make it easier for site operators to negotiate and design pricing. In addition, they plan to provide real-time signals indicating trustworthiness, freshness, and relevance, so AI companies can more easily decide which content they should access.
As for the pricing model, the report doesn't give specific unit prices, and it says custom-made licensing agreements through individual negotiation are the norm. For the market to mature going forward, the report sums up three needs: (1) standardizing bots' self-identification and intent declaration, (2) improving discovery mechanisms that make it easier to find the value of content, and (3) investing in programmable, scalable mechanisms for transactions.
By the way, Cloudflare's own position is remarkable too: more than 20% of the entire web and 36% of the top sites by visits sit behind this network, over 40% of Fortune 500 companies are customers, and it also partners with roughly 80% of major AI companies. It struck me that this is a report only they could write, precisely because it's a place where the data of the search side, the searched side, and the AI-company side all comes together.
Wrap-up
I was amazed by the numbers, generative AI users surpassing 2.5 billion and non-human traffic exceeding 50% for the first time. But what matters even more is that "making things visible" is starting to generate bargaining power and a flow of money. The homework of how to handle mixed-use crawlers still remains, but I think it's a happy change for creators that a mechanism for getting publishers properly paid is gradually taking shape. I'll keep following how it evolves over the coming year!
