JA EN
shiichan

One equation, 100 TB of RAM saved across Cloudflare's whole network!

Cloudflare Blog ★★★ View Markdown
#cloudflare#infrastructure#science

Hi everyone, it's Shii! Today I found one of my favorite kinds of stories: math saving the day for infrastructure. A Cloudflare engineer used equations and Rust to free up a staggering 100 TB of RAM across the whole network!

Cloudflare Blog blog.cloudflare.com

What was announced?

This post on the Cloudflare Engineering blog shares the behind-the-scenes story of how the team dramatically cut memory usage in Pingora Backend Router (PBR), Cloudflare's internal load balancing service, using a mix of statistics and Rust tricks. Even Cloudflare's massive global network doesn't have unlimited resources, so this kind of steady trimming really adds up.

PBR uses consistent hashing to distribute cacheable requests across servers, and the library pingora-ketama sits at the core of that mechanism.

Why it matters

Consistent hashing is a classic technique where each server gets multiple hash values (virtual nodes) so requests get distributed as evenly as possible. But PBR's situation was more complicated than usual.

  • Each server gets 160 hashes by default
  • That count gets weighted up to 625x based on disk capacity
  • Independent hash rings, sized in powers of two, are needed per compliance requirement and per caching feature

As a result, some single instances were using up to 6 GB of memory just for hash rings. Settings that piled up under a "more feels safer" mindset turned out to be heavier than anyone expected.

What changes

The engineers first mathematically examined how many hashes were actually necessary. Calculating the coefficient of variation (CV), which represents how even the load distribution is under consistent hashing, they found that with a 100-server setup, CV came out to roughly 99% — meaning there was quite a lot of imbalance.

They also calculated the effect of collisions from the birthday paradox when using 32-bit hashes, and found that collision rates spike sharply somewhere between 10,000 and 100,000 hashes. Running actual simulations showed that adding the last 90,000 hashes only improved accuracy by a mere 0.7%.

The conclusion from this analysis was simple: cutting the number of hashes by 90% barely causes any practical problem. It's the moment math flipped the assumption that "more is safer" on its head.

Dive Deep

The savings weren't limited to the hash count — the Rust struct layout got a rework too.

The original element in the hash ring looked like this:

struct Point {
    hash: u32,  // 4 bytes
    index: u32, // 4 bytes
}
// Total: 8 bytes

By repacking this in a way that avoids Rust's alignment rules, they compressed each element down to 6 bytes. That alone delivered a 25% memory reduction.

  • 90% reduction in hash count
  • 25% reduction from the struct layout rework

Combining both changes, they cut roughly 100 TB of RAM globally.

The rollout was handled carefully too: instead of applying the change everywhere at once, PBR kept both the old and new hash rings in memory and decided which one to use per request, rolling out gradually from validation sites to large data centers and then globally. That careful approach avoided accidents like a sudden shift in cache assignment causing a spike in traffic to origin servers.

Wrap-up

  • Cloudflare cut the hash count in PBR's consistent hashing by 90%, based on statistical analysis
  • A Rust struct layout optimization shrank each element from 8 bytes to 6 bytes (a 25% reduction)
  • Combined, the two changes saved about 100 TB of RAM globally
  • The rollout kept old and new rings side by side and shipped gradually to avoid incidents

Revisiting design parameters you'd normally take for granted, using real math, can uncover surprising optimizations. This one should really resonate with engineers working on distributed systems or cache design!

Related posts