JA EN
shiichan

Cloudflare cut a 4-hour core server reboot down to 3 minutes!

Cloudflare Blog ★★★ View Markdown
#cloudflare#infrastructure#devops

Hey everyone, it's me, Shii-chan! Today I want to tell you about a mystery where a server reboot suddenly started taking four whole hours. A firmware update made booting slow, and here's how the puzzle got solved!

Cloudflare Blog blog.cloudflare.com

What was announced?

The Cloudflare Blog published the story of how they cut their core servers' boot time from hours down to minutes. Cloudflare's infrastructure splits into the globally distributed edge that handles user traffic, and the core that runs the control plane, billing, and analytics. Today's stars are the bare-metal servers living in that core. After one firmware update, reboots started taking about 4 hours, which turned a fleet-wide rollout that should finish in a day into a multi-day chore. This is the engineering log of how they tracked down the cause and fixed it.

The story so far

The real culprit was that the servers tried every network boot interface one by one. On each boot they went through:

  • IPv4 HTTPS boot (about a 5 minute timeout)
  • IPv4 iPXE (about a 5 minute timeout)
  • IPv6 HTTPS boot (this one finally succeeded)

Every failed attempt waited around 5 minutes for a timeout. As the post puts it:

Every failed network boot attempt burned roughly five minutes waiting for a timeout response.

Four whiffs added up to about 20 minutes per boot, and stacked across a firmware upgrade automation run it ballooned to roughly 4 hours.

What changes

By telling the server the correct boot order up front, the wasteful brute-force search disappeared. The results look like this!

  • Firmware upgrade automation: nearly 4 hours -> 3 minutes
  • A subsequent single boot: about 20 minutes -> under 1 minute

I love how the post wraps it up:

By eliminating the guesswork from our network boot sequence, we turned a four-hour ordeal back into a 3-minute process.

New nodes no longer sink into timeout waits on their first boot, so fleet updates got a lot faster.

Dive Deep

So how did they stop the brute-force search? There were four key moves.

1. Declare the boot order early They reworked the automation to specify the correct network boot interface early in the pre-boot PXE stage. Since a firmware update resets those settings, they also added state validation to reapply the config after an upgrade.

2. Unlock the vendor BIOS The EFI_IFR_REF3 data structure that holds the network boot settings was lazy-loaded, meaning it wasn't created until someone opened the GUI, so automation couldn't see it. Working with the vendor, they enabled specific tokens in the Boot Order Module and got a new BIOS that removed the immutable Force Priority Httpv4 Httpv6 Pxev4 Pxev6 setting.

3. Handle NIC string variations Different NIC vendors name things inconsistently, for example:

UEFI: HTTPS IPv4 Ethernet Network Adapter XXX-XXX-Y for OCP 3.0 P1
UEFI: HTTPS IPv4 Network Adapter - 50:00:E6:8F:4F:32 P1

Exact matching couldn't catch both, so they taught the CfHIIConfig_App tool to use a regex match (.*HTTP.*IPv4.*P1) that absorbs the naming differences.

4. Lighten the iPXE hex comparison They added a boolean flag called uefi-same-hex to detect config changes without running separate show and set commands, trimming the extra variable printing and comparisons during boot.

Wrap-up

  • Cloudflare's core bare-metal servers had degraded to about 4 hour reboots after a firmware update
  • The cause was cycling through network boot interfaces in order, wasting about 5 minutes on each timeout
  • The fix: declaring the correct boot order, unlocking the vendor BIOS, regex-matching NIC strings, and lightening the iPXE comparison
  • Firmware automation went from about 4 hours -> 3 minutes, and a single boot from about 20 minutes -> under 1 minute

If you wrestle with bare metal, PXE boot, or firmware automation, this quiet-but-oh-so-satisfying mystery is for you!

Related posts