Relay + transceiver: peeking into OpenAI's WebRTC redesign for fast voice!

Hey there, it's Shiichan! Today I dug up a wonderfully nerdy infrastructure story about running voice AI at the speed of actual speech. I'm excited!

OpenAI News openai.com

What was announced?

OpenAI's News published a post called "How OpenAI delivers low-latency voice AI at scale." It's an engineering deep-dive on how the team rebuilt the WebRTC stack behind ChatGPT voice and the Realtime API so it stays low-latency even at huge scale.

With voice, even a tiny network hiccup shows up right away as an awkward pause or a clipped interruption, and people notice instantly. So OpenAI had to hit three things at once: global reach for more than 900 million weekly active users, fast connection setup so you can start talking the moment a session begins, and low, stable media round-trip time with little jitter and packet loss.

The story so far

The first implementation was a single Go service built on Pion that did both signaling and media termination. It powers ChatGPT voice, the Realtime API's WebRTC endpoint, and several research projects.

But running that on Kubernetes like a normal service was rough, because WebRTC's one-port-per-session model fits that world poorly. At high concurrency you have to expose tens of thousands of UDP ports, and:

• Cloud load balancers and Kubernetes aren't designed around that many public UDP ports
• A large exposed surface is hard to secure and audit
• Reserving big fixed port ranges fights against pods being added, removed, and rescheduled

On top of that, ICE and DTLS are stateful, so the process that created a session has to keep receiving its packets, or setup breaks. The target was tricky: expose a small, fixed public UDP surface, yet still deliver every packet to the transceiver that owns the session.

What changes

OpenAI's answer is a two-layer relay + transceiver design.

• Relay: a lightweight UDP forwarding layer. It doesn't decrypt media or run ICE state machines. It reads just a bit of metadata and forwards the packet to the right transceiver, keeping the public UDP footprint tiny.
• Transceiver: the stateful endpoint that owns all the WebRTC session state (ICE connectivity checks, the DTLS handshake, SRTP keys, session lifecycle).

The clever part is that clients still see plain, standard WebRTC. Browser and mobile interoperability stay intact, and only the internal packet routing changed. Since most sessions are 1:1 (one user talking to one model), a lightweight transceiver model was the right call instead of a group-oriented SFU.

Dive Deep

The heart of it is how to route that very first packet, without pausing on an external lookup service.

WebRTC already carries a native routing hook: the ICE username fragment, or ufrag, a short identifier exchanged during setup. OpenAI generates the server-side ufrag so it holds just enough routing metadata to infer the destination cluster and owning transceiver.

During signaling, the transceiver returns a shared relay VIP (a virtual IP fronting the relay fleet) and a UDP port in the SDP answer. The client only sees one stable destination like

203.0.113.10:3478

even though many relays sit behind it. The first media packet is usually a STUN binding request, and the relay reads only the ufrag from it to decode the routing hint and forward to the owning transceiver. After that, DTLS, RTP, and RTCP flow within the session without re-reading the ufrag.

The relay keeps deliberately minimal state. If a relay restarts and loses it, the next STUN packet rebuilds it from the ufrag hint. A Redis cache also holds the mapping from the client's IP and port to the transceiver's IP and port, so recovery can happen even earlier.

To take this worldwide, there's Global Relay: geographically distributed relay ingress points so packets enter the network near the user. Signaling uses Cloudflare geo and proximity steering to reach a nearby transceiver cluster, cutting the first-hop latency.

The relay is written in Go and kept deliberately narrow, with no kernel-bypass framework, just a userspace Go process. A few efficiency choices:

• SO_REUSEPORT lets multiple workers bind the same UDP port so the kernel spreads packets across them
• runtime.LockOSThread pins each UDP-reading goroutine to an OS thread, keeping a flow on the same CPU core for cache locality
• Pre-allocated buffers and minimal copying keep GC low

Here's how the post sums up the philosophy:

The broader lesson is that the best place to add complexity is in a thin routing layer, not in every backend service, and not in custom client behavior.

Wrap-up

• OpenAI rebuilt the WebRTC behind ChatGPT voice and the Realtime API into a two-layer relay + transceiver design
• It drops one-port-per-session for a small, fixed public UDP surface that plays nicely with Kubernetes
• Routing metadata rides in the ICE ufrag, so the first packet is steered with no external lookup
• Global Relay plus Cloudflare geo-steering put ingress close to users to cut latency
• The relay is plain Go, and careful SO_REUSEPORT and thread-pinning were enough, no kernel bypass needed

If you run real-time voice or WebRTC at scale, or you've ever wrestled with UDP on Kubernetes, this design write-up will really hit home!