The Internet Now Has a Second Audience: AI Agent Traffic Up Over 1,700%

Hi everyone, it's Shii-chan! Today we're talking about the Internet's "second audience".

What was announced?

The Cloudflare Blog reports that more than half of the traffic on Cloudflare is now automated, and introduces a set of tools for site owners.

The numbers are striking:

Cloudflare handles 115 million HTTP requests per second (up from 63 million at the end of 2024, with peaks above 150 million)
Non-human automated traffic is more than half
AI agent requests grew over 1,700% year over year
In some heavily crawled sectors, human traffic fell by up to 40% in under a year
By June 2026, 52% of crawler requests were for AI training (22% in spring 2025)

When AI gives summaries without sending people to pages, ad and subscription models break: revenue per request falls while costs rise.

Site owners can see, decide, and charge:

See: AI Crawl Control, Business Insights, and BotBase show crawler activity. Web Bot Auth cryptographically verifies agent requests from operators like OpenAI, Google, and AWS. Cloudflare Radar tracks crawlers publicly
Decide: Separate Search, Agent, and Training controls are available on all plans including Free. "Disallow AI Training" keeps a site indexed while blocking training use, and Apple, Google, and Microsoft have committed to honoring it (shipped September 15)
Charge: Pay Per Use (beta) and Monetization Gateway (closed beta for eligible U.S. customers)

Monetization Gateway prices per request, query, or token using HTTP 402 Payment Required and the open x402 standard
Markdown for Agents serves a stripped-down version that is easier for bots to read
WebMCP exposes site actions directly to agents
A joint research pilot with OpenAI aims to make content discovery more efficient
There are over 500 billion verified bot requests per week, and 50+ publisher-AI licensing deals since 2023

This is for site owners deciding how to handle AI access!