Cloudflare's security scans got 10x faster — 120 scans per second with zero extra hardware!

Hey everyone, it's Shii-chan! Today I've got a really meaty behind-the-scenes story: Cloudflare's engineers made their security scanning 10x faster, without adding any hardware. Pulling that off with just smarter code and queries? That gets me excited!

Cloudflare Blog blog.cloudflare.com

What was announced?

This is an engineering post from the Cloudflare Blog explaining how the team scaled their Security Insights system.

Security Insights regularly scans Cloudflare accounts to find security risks and misconfigurations. Under the hood, it streams events through Apache Kafka, uses Go microservices called 'checkers' to scan specific assets, and sends the results to an internal API backed by Postgres.

Since it's an engineering post, it's less of a launch announcement and more of a deep technical look at what was clogged up and how they fixed it.

The story so far

The old setup had three pretty big pain points:

• Scans only ran once every 1–2 weeks, so security gaps went unnoticed in between
• Automatic scanning was opt-in for free accounts, leaving many of them unprotected
• The infrastructure was already straining: millions of events piled up in the backlog, the API frequently timed out, and processes sometimes crashed

The goal was clear: push throughput from 10 scans per second to 100 — a 10x jump — and do it without adding hardware.

What changes

After all the tuning, scan frequency improved a lot:

• Free: every 7 days
• Pro / Business: every 3 days
• Enterprise: daily

And it now covers all customers, so accounts that used to be thinly protected are included too. On top of that, the team added on-demand scans you can trigger at the account, zone, insight, and insight-type levels.

Dive Deep

Here's the fun part! Let's walk through the four places they cleared the bottleneck.

Running Kafka in parallel

Adding partitions would have burdened shared infrastructure, so instead they leaned on parallel processing. Checkers now receive messages in batches and process each one in its own goroutine. They also split 'slow lane' and 'fast lane' consumer groups so slow messages don't hold up the fast ones.

Batching database queries

The original code ran an individual INSERT for every single insight. A batch could contain up to 500,000 insights, which meant half a million round trips for a single API call — heavy!

So they switched to a two-pronged approach: UNNEST for smaller datasets (done in milliseconds) and COPY for larger ones (huge sets processed in seconds).

Fixing API latency

The API ran active-active across Portland and Amsterdam, and that setup was producing latencies over 50 ms. In the team's words:

Our average API call completed in 10 ms in Portland, but almost 3 seconds in Amsterdam!

That was exhausting the connection pools and causing timeouts. Switching to active-passive routing, aligned with the primary Postgres in Portland, cleared up the latency overnight.

Rebuilding the scheduler

The old scheduler distributed scans unevenly, and delays cascaded for large accounts. Three changes fixed it:

• Scheduling each zone independently with its own timestamp
• Randomizing existing last_scheduled_at values to smooth out the skew
• Adding adaptive rate limiting that recalculates every 30 minutes based on account counts and scan frequencies

Wrap-up

• This is the behind-the-scenes story of how Cloudflare scaled the Security Insights scanning platform 10x with no extra hardware
• At peak it now sustains over 120 scans per second, clearing the original 10x goal
• Parallelizing Kafka, batching queries with UNNEST / COPY, moving to active-passive, and rebuilding the scheduler all did the heavy lifting
• Scan frequency dropped to every 7 days for Free, every 3 days for Pro / Business, and daily for Enterprise — and now covers all customers
• A great read for backend and infrastructure engineers who love beating bottlenecks with smarter code and queries rather than more hardware!