Taming 2 billion events per second: how Cloudflare keeps pipeline hardware down

Hi everyone, it's Shii-chan! Today's story is how Cloudflare's Data team fights ever-growing data without just adding hardware!

Cloudflare Blog blog.cloudflare.com

What was announced?

The Cloudflare Blog describes how the Data team has optimized its analytics and log data pipeline. In May 2026 the pipeline handled about 2 billion events per second. The team says its changes over five years are equivalent to deferring about 2.5 years of hardware expansion.

Why it matters

Traffic grows exponentially, while hardware grows roughly linearly. So the team has repeatedly cut the data volume it must carry.

What changes

Three changes in 2026 alone eliminated about 70 GB/s of compressed backbone traffic combined. Throughput is about eight months behind its previous growth projection, and the team estimates its projected 2031 data volume has been halved.

Dive Deep

The pipeline looks like this:

• Logfwdr: collects data from distributed sources such as Workers and FL
• Centralized receivers: receive the data from Logfwdr
• Buftee: the buffering system the receivers write to
• Consumers: read from Buftee and deliver to destinations such as ClickHouse

Here are the optimizations, year by year:

• 2022, schema split: a single monolithic schema became product-specific schemas. Some fields are duplicated, but backbone bandwidth fell 35-40%
• 2024, compression: LZ4 was replaced with zstd. CPU impact was negligible, and compressed throughput fell nearly 40%, so the same pipe supports over 1.5x the traffic
• 2026, load balancing: Logfwdr used to send two full copies of each non-regional event, one per region. Now it sends each event once to any available receiver, and replication to the second region happens later from Buftee. The net reduction is 36 GB/s
• 2026, sampling at the edge: records are shuffled per key, then a sample of size √N is taken from N records. A key with 100x more data stores about 4x more rows
• Fall 2026, aggregation in the pipeline: rollups now run at four stages: Logfwdr, the receivers, the inserters, and ClickHouse aggregating table engines. HTTP rollup traffic fell from over 9 GB/s to under 2 GB/s, about 80%

Load balancing works because sampling now happens before buffering, so less data is replicated, and because streams are partitioned into uniform batches that compress better.

One note: the original post gives the load-balancing net reduction as both 36 GB/s and about 38 GB/s.

Wrap-up

• The team keeps reducing data volume on a pipeline of about 2 billion events per second
• Steps include schema split, zstd, load balancing, edge sampling, and in-pipeline aggregation
• The 2026 changes eliminated about 70 GB/s of backbone traffic
• The aggregation work also led to a product, Ready-Aggregates

This one is for anyone who designs large data pipelines or wants ideas for cutting costs!