MRC: Wiring About 131,000 GPUs With Just Two Switch Tiers!

Hey everyone, it's Shii-chan! Today we're diving into the world of supercomputer networking, the cabling that ties tens of thousands of GPUs together. It usually stays behind the scenes, but the tricks here are so clever I couldn't stop reading!

OpenAI News openai.com

What was announced?

OpenAI, together with AMD, Broadcom, Intel, Microsoft, and NVIDIA, announced MRC (Multipath Reliable Connection), a new networking protocol for large AI training clusters. This one comes from OpenAI's News. And instead of keeping the spec to themselves, they released MRC through the Open Compute Project, so it can become a shared standard everyone can build on.

Why it matters

When you train an AI model, huge numbers of GPUs have to move data in lockstep. If even one transfer arrives late, the delay ripples through the whole job and GPUs sit idle. And the bigger the cluster, the more routine network failures become: a single failure used to crash an entire training job, or stall it for seconds. Congestion, link failures, and device failures are the most common sources of delay.

What changes

With MRC, GPUs keep moving even when things get congested or break. During real training runs, links between tier-0 and tier-1 switches flapped multiple times a minute, yet MRC left no measurable impact. Rebooting a tier-1 switch used to need careful coordination with the training teams, but with MRC they didn't even have to coordinate.

Dive Deep

There are three big ideas.

First, multi-plane networks. One 800Gb/s interface is split into eight 100Gb/s links to eight switches, creating eight parallel planes. That lets you connect about 131,000 GPUs with just two tiers of switches, instead of the three or four tiers a conventional design needs. Fewer tiers means less power and fewer components.

Second, adaptive packet spraying. Instead of pinning a transfer to one path, MRC sprays its packets across hundreds of paths spread over all the planes. Packets arrive out of order, but each carries its destination memory address so it lands in the right place. It also uses packet trimming to cut down false-positive congestion signals.

Third, source routing with SRv6. Instead of dynamic routing like BGP, the sender writes switch identifiers into the packet header to pick the exact path. Switches just use a static routing table set once at startup. That's why MRC can detect a failure and reroute on a microsecond timescale, versus seconds or tens of seconds on conventional fabrics.

Under the hood it extends RDMA over Converged Ethernet (RoCE) and borrows techniques from the Ultra Ethernet Consortium. If you want the deep details, read the technical paper Resilient AI Supercomputer Networking using MRC and SRv6.

MRC is already running on all of OpenAI's largest NVIDIA GB200 supercomputers. As part of Stargate, it powers sites like Oracle Cloud Infrastructure (OCI) in Abilene, Texas and Microsoft's Fairwater supercomputers, and it has already trained multiple OpenAI models.

Wrap-up

• MRC is a new networking protocol OpenAI built with five partners for large-scale AI training, released through the Open Compute Project
• The multi-plane design connects about 131,000 GPUs with only two switch tiers, saving power and components
• Packet spraying across hundreds of paths nearly eliminates core congestion and throughput variance
• Static SRv6 routing detects failures and reroutes in microseconds, so flapping links don't stall training

If you work on networking or infrastructure, or you're curious about running massive training jobs, this is a treat. It's not something you'll spin up over the weekend, but the design philosophy for keeping tens of thousands of GPUs running without stalls is packed in here, worth the read on its own!