Cloudflare is rethinking CDN cache for the AI era!

Hi there, I'm Shiichan! Today's topic is caching, and honestly I'm a little giddy about it. The quiet workhorse that keeps the web fast is being rethought from the ground up because of AI!

Cloudflare Blog blog.cloudflare.com

What was announced?

The Cloudflare Blog published a piece arguing that we should rethink CDN cache design from scratch for the AI era. The trigger: 32% of the traffic across Cloudflare's network now comes from automated traffic (bots, AI crawlers, and the like). That traffic behaves nothing like human browsing, so today's caching approach is starting to struggle.

So Cloudflare is exploring new cache replacement algorithms, and even separate cache tiers for AI versus human traffic.

Why it matters

A cache is how a CDN keeps copies of popular pages close by. If the content is on hand (a cache hit) it's fast and cheap to serve; if it isn't (a cache miss) the request travels all the way to the origin server. So the whole game is: how much of the popular stuff can you keep nearby?

AI crawlers flip that on its head. Here's how the post puts it:

This repeat access to long-tail assets churns the cache that the human traffic relies on.

• They hit different URLs every time, so their unique access ratio is very high, typically 70% to 100%
• Instead of popular pages, they dig deep into rarely read, long-tail corners of a site
• Their URL handling is sloppy, so a substantial fraction of fetches end in 404 errors or redirects

Because of this, pages that used to be "long-tail" are suddenly requested a lot, pushing out the content humans actually need. Real reports include Wikipedia seeing a 50% surge in multimedia bandwidth from bulk image scraping, SourceHut facing service instability, and Read the Docs having large files downloaded hundreds of times a day.

What changes

Cloudflare's idea is to split the paths for humans and AI.

Even within AI traffic, latency-sensitive work like RAG (retrieval-augmented generation) or real-time summarization should go to caches that balance larger capacity with moderate response times.

Meanwhile, work that isn't in a hurry, like bulk scraping for model training, can be served from deeper, higher-latency cache tiers, or even delayed with queue-based admission and rate-limiters so the backend doesn't get overloaded. When infrastructure is under load, bulk scraping can be deferred without hurting human visitors.

Dive Deep

Cloudflare's cache today is managed with LRU (least recently used, evict whatever was used least recently first). But when a crawler floods it with one-shot accesses, content humans still need gets evicted.

As a near-term experiment, Cloudflare is testing newer replacement algorithms like SIEVE and S3FIFO. According to the post, these could let human traffic keep the same hit rate with or without AI interference. They're also researching workload-aware, machine learning-based caching.

Long term, they think the best path is a dedicated cache layer for AI, routing humans and AI into distinct tiers.

Cloudflare also already ships tooling for this: AI Index to bundle up a site's content, Markdown for Agents for lightweight agent-friendly pages, AI Crawl Control to manage crawlers, and Pay Per Crawl to charge for crawling. Rethinking the cache is the next step in that story.

Wrap-up

• 32% of network traffic is automated, and AI crawlers do the opposite of humans: hitting long-tail content with unique URLs every time
• That drives up cache misses, with real impact at Wikipedia, SourceHut, and Read the Docs
• Near term: moving from LRU toward SIEVE / S3FIFO and ML-based caching; long term: a separate cache layer for AI
• Non-urgent training crawls get absorbed by deep tiers plus queues and rate limits

If you love the guts of CDNs and caching, or you run a site that's getting its bandwidth eaten by AI crawlers, this one's for you!