Cloudflare and OpenAI Team Up: ChatGPT Search Found Hundreds of Millions of New Pages!

Hi everyone, it's Shii-chan! Today I'm sharing a story about how Cloudflare and OpenAI worked together on AI content discovery. It's exciting news!

Cloudflare Blog blog.cloudflare.com

What was announced?

On the Cloudflare Blog, Cloudflare and OpenAI shared progress on a research pilot. It started in July and tests whether signals from Cloudflare's network can help ChatGPT users find new content more efficiently.

Early results are positive. Previously unknown pages were added to OpenAI's search index, which gives ChatGPT more sources to draw on!

Why it matters

Content is only useful to AI if the AI can find it. Pages like a product manual, passport requirements, or a restaurant's current menu can be very valuable even when few other sites link to them.

Traditional search ranked pages and left people to judge the information themselves. AI systems have to find the right sources, judge how useful they are, and connect people to them through answers, citations, or visits. That makes discovery quality important.

What changes

• Website owners: less unnecessary crawler load, and better discovery for content you choose to make available to AI
• OpenAI: more of its search resources go toward information that helps people
• ChatGPT users: more relevant, up-to-date answers with links to sources

ChatGPT Search links back to the original sources, so visits to publishers remain possible.

Dive Deep

Signals from Cloudflare's network help OpenAI's search crawler find more useful pages. OpenAI independently evaluates and ranks them for relevance and helpfulness. The authors describe this as a feedback loop that can improve both discovery and information quality.

The pilot covered about 1% of Cloudflare's network, and OpenAI discovered and indexed hundreds of millions of new web pages.

There are three main building blocks:

• Filtering before fetching: patterns that waste crawler effort, such as bots requesting files that don't exist, sites that redirect almost any address to the homepage, and different addresses serving duplicate content, can be recognized so crawlers skip them before downloading
• Harm flagging (experimental): potentially harmful pages, like a fake login form planted on a legitimate site, are flagged at the page level rather than the whole domain. That protects users without penalizing the site owner. Early results suggest it can identify pages that should be kept out of the index
• Change signals: they tell crawlers when a page is worth revisiting, so unchanged pages aren't downloaded again

The post is also upfront about a trade-off. Discovering new pages may increase crawling at first, because new pages must be evaluated and indexed. The goal is to keep discovering and staying current while lowering total crawling work over time.

On permissions and data use: a signal about a page does not give permission to access it. The research uses web metadata for search and discovery, not page content for model training. Site owners can manage how AI systems access their sites with Cloudflare's AI Crawl Control.

Next, the partners plan to improve page selection, test faster and more reliable change signals, and measure what happens after content is discovered. They also plan to study how better discovery can translate into business outcomes.

Wrap-up

• Cloudflare and OpenAI ran a research pilot that uses network signals to help ChatGPT search discover pages
• On about 1% of the network, hundreds of millions of new pages were indexed
• Three pillars: skipping wasteful crawls, page-level harm flagging, and change signals
• Signals are not access permission, and page content is not used for training
• Site owners can manage AI access with AI Crawl Control

This one is for site owners worried about crawler load, and for anyone who wants their content to be found by AI search!