Code Orange: Fail Small is done — Cloudflare's network now breaks small!

Hey everyone, it's me, Shii-chan! Today we're talking about how to make infrastructure fail smarter. It sounds a little nerve-wracking, but this is the quiet, super-important kind of work that keeps everything running.

Cloudflare Blog blog.cloudflare.com

What was announced?

On the Cloudflare Blog, there's a completion report for a two-quarter, all-hands engineering initiative called 'Code Orange: Fail Small'. It started in response to the big outages in November and December 2025, and its whole goal is to make the network much harder to break in a big way. The original plan was shared in the Fail Small resilience plan, and this post is the 'we did it' wrap-up.

'Fail Small' means exactly that: instead of trying to make failures impossible, you make sure that when something does break, the blast radius stays tiny.

The story so far

The trigger was two major outages: the November 18, 2025 outage and the December 5, 2025 outage, where a configuration change spread across the whole network in an instant.

Take the November Bot Management classifier issue. When config is pushed everywhere at once, a bad change reaches production just as fast as a good one. Being 'fast' had quietly become a risk.

What changes

From now on, configuration changes don't go out 'all at once'. They roll out gradually while health is being watched, and if something looks wrong, it rolls back automatically. So even if that same Bot Management issue happened again, it would be caught in the earliest stage of the rollout, affecting only a small slice of traffic before being reverted.

Incident communication changes too. A dedicated communications team works alongside the responders, and during an active incident they've committed to updates every 30-60 minutes. The goal is to tell customers before they even notice.

Dive Deep

Several concrete pieces showed up here:

• Snapstone: a new internal component for 'health-mediated deployment' of config changes. It rolls things out progressively while watching health in real time, and rolls back automatically when needed.
• Engineering Codex: a mandatory internal ruleset for every team, enforced by AI code review. It captures best practices — for example, a rule not to use .unwrap() outside of tests and build.rs — and catches violations at the merge-request stage instead of in production.
• System segmentation: critical services like the Workers runtime are split into independent segments serving different customer cohorts, with staged rollouts that start with free customers first. In one seven-day window, more than 50 deployments rippled across edge segments in coordinated 'waves'.
• Break Glass procedures: backup authorization paths, emergency scripts, and proxies for 18 key services. On April 7, 2026, they even ran a company-wide drill with over 200 people.

Failure modes are designed on purpose too: depending on how critical a service is, it can 'fail stale' (keep using the last known-good config) or 'fail open/close'. This 'change safely at any scale' mindset ties back to Safe change at any scale.

Wrap-up

• The two-quarter 'Code Orange: Fail Small' initiative is complete, aiming for a network that fails small
• Snapstone gives staged config delivery plus automatic rollback; the Engineering Codex is enforced by AI review
• Critical services like Workers are segmented and roll out to free customers first
• 18 services have Break Glass procedures, backed by a company-wide drill with 200+ people
• A dedicated team handles incident comms with updates every 30-60 minutes

As a real example of pivoting from big outages to breaking small, this one really lands for SREs, reliability-minded folks, and anyone who designs large-scale deployments!