Bot and crawler caching is back! Cloudflare fixes a quiet APO regression
Hello, it's me! Today's a fix that you probably didn't notice was costing you something. Let's dig in!
Cloudflare Changelog
developers.cloudflare.com
What was announced?
According to the Cloudflare Changelog, a regression in Automatic Platform Optimization (APO) has been fixed.
APO caches HTML pages at the edge to speed up delivery, mainly for platforms like WordPress. But requests that didn't send an explicit Accept: text/html header were being served straight from origin (cf-cache-status: DYNAMIC) instead of from cache.
The story so far
Requests without an explicit Accept: text/html header — or that just send a wildcard value — aren't unusual. Common sources include:
- Search engine and other crawlers
- Various bots
- Uptime monitoring services
These requests should have been cached but were falling through to origin instead. That meant extra load on origin servers and a lower-than-expected cache hit rate for affected sites. Some users had worked around it by adding a Transform Rule that forced an Accept: text/html header.
What changes
With the fix in place, APO now correctly serves cached responses to these crawlers, bots, and monitoring requests again.
- Less unnecessary load on origin servers
- Faster responses for crawler and bot traffic
- If you added a workaround Transform Rule, you can remove it and let APO handle it natively
Wrap-up
- APO had a regression that skipped caching for requests without an explicit
Accept: text/htmlheader - Crawlers, bots, and uptime monitors were the main ones affected
- The fix restores caching for these requests; no action is needed
- Any workaround Transform Rule setting
Accept: text/htmlcan be safely removed
A small but welcome fix for anyone running APO or running a site that gets a lot of crawler and bot traffic!
