JA EN
shiichan

When idle isn't idle: the death spiral that trapped quiche's CUBIC!

Cloudflare Blog ★★★ View Markdown
#cloudflare#infrastructure#web

Hey everyone, it's Shiichan! Today we're going deep into the guts of networking — a congestion-control bug hunt. It's such a cool investigation, so let's dive in together!

Cloudflare Blog blog.cloudflare.com

What was announced?

On the Cloudflare Blog, the team published the story of a bug hidden in quiche (Cloudflare's open-source QUIC / HTTP3 implementation). It's titled "When idle isn't idle." After heavy packet loss, CUBIC's congestion window (cwnd) would get pinned at its minimum and never recover — a "death spiral." It all started when an internal ingress-proxy integration test began failing 61% of the time: even with a 10-second timeout, around 60% of the tests couldn't finish the download.

The story so far

CUBIC, standardized in RFC 9438, is the default congestion controller in Linux, so it governs how most TCP and QUIC connections on the public Internet behave.

The story begins with the Linux kernel in 2017. When an app finished sending and went quiet (idle), CUBIC's epoch (the start point of its growth curve) wasn't updated, so time kept passing. On resume, now - epoch_start grew huge and the window ballooned to an unrealistic value. Eric Dumazet, Yuchung Cheng, and Neal Cardwell fixed it with an elegant change: instead of resetting the epoch, shift it forward by the idle duration — keeping the curve's shape and only sliding it in time. In the kernel, a CA_EVENT_TX_START callback told real idleness apart from a brief pipeline stall.

What changes

With the fix, CUBIC recovers correctly even after brutal packet loss, and the window grows back along the real CUBIC curve. All tests pass, and downloads finish in a few seconds. Anything running quiche (including Cloudflare's edge!) is rescued from the "throughput mysteriously won't grow" problem on lossy links like mobile networks.

Dive Deep

CUBIC was ported to quiche back in 2020. The idle adjustment came along for the ride, but QUIC runs in userspace, not the kernel, so there's no CA_EVENT_TX_START callback. Instead, quiche checked for idleness inside on_packet_sent() by asking "is bytes_in_flight == 0?"

if bytes_in_flight == 0 {
    let delta = now - self.last_sent_time;   // this is the trap
    self.congestion_recovery_start_time += delta;
}

That was the trap. Once cwnd collapsed to its minimum (two packets, 2700 bytes) after loss, it fell into this loop:

  • Send the two packets you have
  • About one RTT (~14 ms) later, both are ACKed and bytes_in_flight drops to zero
  • On the next burst, on_packet_sent() sees bytes_in_flight == 0
  • It computes delta = now - last_sent_time — but last_sent_time is from the start of the previous RTT, so delta becomes a full RTT (~14 ms), not the near-zero real idle gap
  • The inflated delta pushes congestion_recovery_start_time into the future
  • With the recovery start time in the future, in_congestion_recovery() keeps returning true, so CUBIC thinks it's still recovering and skips growth
  • cwnd stays at two packets; the next ACK drains the pipe, and back to the top…

So cwnd stayed pinned at the minimum while the state bounced between recovery and congestion avoidance every ~14 ms — 999 transitions in all. Triggering the trap required three things at once: (1) a real loss event had set the recovery boundary, (2) the connection was in congestion avoidance, and (3) cwnd was at the two-packet minimum. That's why Reno and higher-window connections never showed it.

Here's the fun part: about a week after their first fix, the Linux kernel team spotted the same weakness themselves and landed a follow-up fix — measuring idle from send events could fling epoch_start into the future, the very same pitfall.

The fix is small: remember the ACK arrival time last_ack_time, and measure the idle start from "when bytes_in_flight actually hit zero" rather than "the last send."

if bytes_in_flight == 0 {
    let idle_start = max(last_ack_time, last_sent_time);
    let delta = now - idle_start;
    congestion_recovery_start_time += delta;
}

With max(last_ack_time, last_sent_time), at the minimum window delta becomes near-zero so the recovery boundary doesn't jump into the future. For genuinely long-idle connections, last_ack_time stays far in the past, so Dumazet's "shift the epoch forward" behavior is preserved too. Both cases handled — so satisfying!

Wrap-up

  • quiche's CUBIC pinned its congestion window at the minimum (two packets / 2700 bytes) after heavy loss, oscillating through 999 state transitions every ~14 ms — a "death spiral"
  • The cause: the userspace idle check used now - last_sent_time, wrongly adding a whole RTT instead of the real idle gap
  • That pushed the recovery start time into the future, so CUBIC thought it was "still recovering" and kept skipping growth
  • The fix adds last_ack_time and measures from when bytes_in_flight truly hit zero; tests went to 100% passing
  • A treat for low-level fans who love tracing QUIC and congestion control, and for network engineers battling "throughput won't grow on lossy links"

Related posts