Cloudflare's Cache Runs on Pingora Now!

Hey there, it's Shiichan! Today's a bit of a behind-the-scenes story: Cloudflare swapped out the whole engine under its cache.

Cloudflare Changelog developers.cloudflare.com

What was announced?

Over on the Cloudflare Changelog, Cloudflare announced that its cache now runs on a new proxy built on Pingora, a Rust-based framework. Pingora already serves a significant portion of Cloudflare's network traffic, and now there's a cache-focused proxy built on top of it. The new proxy is faster, more memory-safe, and designed so the cache architecture can keep evolving.

The story so far

The previous cache ran on a different stack, and adding features or matching the finer points of the RFCs had been getting harder over time. Moving onto the Rust-based Pingora cleans up the foundation and lifts both performance and standards compliance a notch.

What changes

• Lower latency, thanks to less per-request overhead and better connection reuse.
• Fewer cache MISSes from improved cache retention, which also lightens the load on your origin (origin offload).
• Caching behavior that follows the HTTP standards more closely.
• And most importantly, a foundation ready for upcoming cache features.

Dive Deep

Two new features are highlighted.

First, asynchronous stale-while-revalidate. Instead of the first request after expiry blocking on the origin, every request now returns stale content right away while revalidation runs in the background. See the asynchronous stale-while-revalidate changelog for the details.

Second, unbuffered bypass by default. Responses that bypass the cache are streamed straight to the client without buffering, cutting the time-to-first-byte for uncacheable content.

There are also some behavioral changes for RFC compliance and correctness, and these can touch existing setups, so they're worth a look.

• Responses with Vary: * now bypass the cache. Per RFC 9110 Section 12.5.5, a * means the response varies on factors beyond request headers and must not be served from cache, so Cloudflare bypasses it instead of storing it.
• For cacheable assets, Set-Cookie is now stripped on MISS and EXPIRED responses, not just on HITs.
• Floating-point TTLs like max-age=1.5 used to be rejected as invalid; now they round down to the nearest integer.

A deeper write-up is coming to the Cloudflare blog soon. For background on Pingora itself, Open sourcing Pingora and How we built Pingora are great reads.

Wrap-up

• Cloudflare's cache migrated to a new proxy built on the Rust-based Pingora.
• The goals: lower latency, fewer MISSes, better RFC compliance, and a base for future features.
• New features: asynchronous stale-while-revalidate and unbuffered bypass.
• Behavioral changes: Vary: * bypass, Set-Cookie stripping on MISS/EXPIRED, and floating-point TTL rounding.

You don't have to configure anything to get the speed-up, but if you fine-tune your cache headers, skim the behavioral-changes section just to be safe.