Half the DNS build delay and 85% less memory: Cloudflare's smallest-safe-build planner

Hi everyone, it's Shii-chan! Today I'm sharing a story from behind Cloudflare's DNS: how build delay and memory use dropped a lot!

Cloudflare Blog blog.cloudflare.com

What was announced?

The Cloudflare Blog described "Targeted Zone Builds" for the DNS Zone Builder. Instead of rebuilding a whole zone on every zone setting change, Cloudflare built a planner that picks the smallest safe build.

The results:

• Full builds fell by about 40% (from about 11.88% of builds to 7.04%)
• Peak memory fell from 79 GiB to 12 GiB (about 85%)
• Median build completion delay fell from about 7.7 seconds to about 3.3 seconds

The story so far

The Zone Builder turns DNS changes into data that authoritative DNS servers can use, and writes changed keys to Quicksilver. It handles about 700 DNS record changes per second (up from about 250 in 2022), plus about 100 other events such as zone setting changes.

The pain points were:

• A zone setting change could trigger a full build: read every record, convert it to Quicksilver keys, compare with the stored keys
• A full build held the whole zone in memory, and zones with more than 20 million records sometimes did not fit at all
• A long full build held back Kafka commits for its partition, delaying unrelated changes

What changes

Now only the necessary work is done for each kind of change. In a 24-hour measurement, builds broke down like this:

• Resources only: 86.81%
• Resources and metadata: 2.54%
• Apex and metadata: 2.30%
• Purge: 1.30%
• Full zone: 7.04%

Most builds are small, so setting changes on huge zones no longer blow up memory, and other zones wait less. The largest p99 tail-delay spike dropped from 14.6 minutes to 1.21 minutes, about 12 times lower!

Dive Deep

How the planner thinks

Events now carry the changed fields with previous and new values. Timestamp-only updates, unknown events, and events with missing details still trigger full builds. In the post's words, "the planner never turns uncertainty into less work."

There are five build types:

• Resources only: rebuild specific DNS names (ordinary A, AAAA, TXT, CNAME changes)
• Resources and metadata: rebuild names plus shared metadata (nameserver and wildcard changes)
• Apex and metadata: rebuild the zone apex and metadata (for example, Universal SSL)
• Purge: remove a zone that should no longer be published
• Full zone: rebuild everything when smaller work can't be proven safe

The rule: every key a targeted build writes must match, byte for byte, what a full build would write. If that can't be proven, the build is full. For rebuilt names, the builder scans Quicksilver for existing keys and deletes any the new build didn't write.

Two-stage planning and a unified scheduler

The event creates a conservative work list. Right before execution, the builder reads fresh state from database replicas (settings, zone membership, same-name zones). That can upgrade a build to full, convert it to a purge, or skip it. A final check rejects reads from lagging replicas.

One scheduler replaces the separate record and full-zone schedulers and merges queued work for the same zone. Any full requirement makes the merged build full, and a purge overrides everything. Kafka messages stay attached to the work until results are written, so retries don't lose changes.

Kafka soft commits and fairness

Builds can finish out of order in memory, but the committed offset advances only after all earlier messages finish. After a restart, unfinished messages are replayed; the team chose repeated work over lost changes. Each zone has a per-plan rebuild interval, and changes arriving in that window merge into the next build.

Handling huge zones

Above a record-count threshold, the builder never escalates to a full build. It rebuilds the apex, metadata, and queued names instead, because a full build of those zones can't finish. The team accepts some same-name conflict risk, and rare cases can be fixed by rebuilding names manually.

Testing and rollout

• Equivalence tests compare full and targeted builds from the same database state, key by key
• Shadow comparison ran a full build beside each targeted build during rollout
• A Mirror Zone Builder wrote production events to a separate non-serving Quicksilver and compared 20% of zones (0.31% diff rate across 234,313 matched zones, mostly timing-related)
• Gradual rollout by hashing the zone name, Free zones first, then paid

Caveats

• Full builds are still needed for first publication, changes that can affect any record, and events that don't identify their changes
• Latency figures are not guarantees: they combine all build types and plans and exclude Quicksilver's network distribution
• Much of the delay comes from rebuild intervals, not build work

Wrap-up

• Cloudflare replaced "rebuild the whole zone" with a planner that picks the smallest safe build
• Full builds fell about 40%, peak memory about 85%, and median build completion delay went from about 7.7 s to about 3.3 s
• The design is safety-first: if key equality can't be proven, it falls back to a full build
• Shadow comparison and a Mirror Zone Builder validated the rollout step by step

This is for anyone interested in large-scale data pipelines and in moving to incremental processing without losing safety!