OpenAI's New Voice System GPT-Live Keeps the Conversation Flowing Without Turn Detection!

Hi everyone, it's me! Today I'm walking you through an engineering story from OpenAI about how they rebuilt the guts of ChatGPT Voice from scratch. It might sound like a backend deep dive, but trust me, it's a good one, so stick with me!

OpenAI News openai.com

What was announced?

OpenAI's News shared the story behind GPT-Live, the third-generation voice system that powers ChatGPT Voice. GPT-Live removes the turn detector from the audio path entirely, something earlier voice AI systems relied on.

The voice model itself is now full-duplex, meaning it can listen and speak at the same time. When deeper reasoning or tool use is needed, it can consult frontier models like GPT-5.5 without interrupting the flow of conversation. This foundation also powers the newly launched ability in the ChatGPT desktop app to control your computer and coordinate your agents.

The story so far

Earlier voice AI systems inherited the turn-based nature of text LLMs, just with each turn represented as an audio blob instead of text. The system stayed turn-based no matter how the pipeline was built:

• Cascaded systems: speech-to-text, the LLM, and text-to-speech ran in series. That sequencing added latency and lost cues like tone and pacing
• Speech-to-speech models: skipping the transcription step helped, but the system still relied on a turn detector to decide when inference could begin, so the interaction stayed turn-based

In other words, no matter how smart the model got, the accuracy of a small model deciding whether the other person had finished speaking set the pace of the whole conversation. Guess too soon and you cut the user off; guess too late and the response feels sluggish.

What changes

With GPT-Live, the voice model itself stays in control of the conversation. Audio flows continuously in and out of the model while deeper reasoning and tool use happen asynchronously behind the scenes. The system's top priority is keeping the media loop uninterrupted, while heavier work like invoking frontier models and persisting the conversation happens off the live path.

This doesn't just make turn-taking feel more human. It's also the foundation that lets ChatGPT Voice grow from "just talking" into controlling your computer and coordinating your agents. Down the line, this same architecture will underpin an upcoming GPT-Live API.

Dive Deep

Here's the more technical side, some of what OpenAI reworked over six months.

Rewriting the media frontend in Go

The media frontend and inference logic used to be written in Python's asyncio. OpenAI rewrote them in Go, and the post states that the new system's p95 latency now matches the previous system's p50. In other words, what used to be the median speed is now achievable even at the 95th percentile, a substantial improvement.

Seamless handoffs between model instances

In a long voice session, context keeps growing while model instances spin up and down based on demand. OpenAI built a handoff mechanism that warms a replacement model instance alongside the existing one, prefills it with the current session context, runs inference against both in parallel, and cuts over once the new instance is ready. The same mechanism supports context compaction: as context approaches its limit, the system compacts it and prepares a replacement instance in the background, without ever interrupting the audio.

Delegating to GPT-5.5 at conversational speed

When GPT-Live hands off deeper reasoning or tool use to a frontier model like GPT-5.5, several tricks keep latency down:

• An inference session for the frontier model is created and prefilled with the initial context as soon as the voice session starts
• That session stays alive for the whole conversation, using stable session affinity for successive requests
• Prompt caching further trims latency
• Reasoning effort, output limits, tool schemas, and model-tool round trips are all tuned for faster responses

Turning continuous speech into discrete turns

The voice model treats the conversation as one continuous stream, but the surrounding systems, including ChatGPT's conversation UI, analytics, and safety infrastructure, still need discrete user and assistant turns. The application server infers who's speaking from partial transcripts and timing signals, and keeps two views of the conversation at once: a speculative view that can still change, and an authoritative record that's finalized. The UI uses the speculative view since it can handle updates; analytics logging uses the finalized one.

Starting sessions faster with WARP and Instant Connect

WebRTC is a strong foundation for low-latency media, but a vanilla WebRTC session needs a surprising number of handshakes before it can start. OpenAI worked with the WebRTC community to design WARP, an open specification now advancing through the IETF's TSVWG working group, with support already added to libwebrtc and Pion.

They also built Instant Connect, which negotiates SDP parameters ahead of time so that exchange doesn't sit on the critical path. Combined, WARP and Instant Connect let a client start a session with a single UDP packet.

Silent-testing in production

Before letting GPT-Live talk to real users, OpenAI ran a silent test that routed a small, gradually increasing share of production ChatGPT Voice sessions to the new system in read-only inference mode. A few lessons stood out:

• Capacity couldn't be reduced to GPU throughput alone; CPU-side stream handlers, queues, and network paths all needed to scale together
• Routing sessions to distant capacity added delay, making geography a first-order concern
• Problems like memory pressure in long sessions, compaction and state restoration on reconnects, and races in the shutdown handshake on disconnects only showed up over realistic session lifecycles, not in short load tests

In response, the team added more granular telemetry, validation against known-good configurations, staged rollouts, and the ability to quickly isolate unhealthy paths.

Wrap-up

• OpenAI detailed GPT-Live, the third-generation voice system behind ChatGPT Voice
• It removes the turn detector from the audio path, making full-duplex "listen while speaking" conversation possible
• Deeper reasoning and tool use are delegated asynchronously to frontier models like GPT-5.5
• The media frontend was rewritten in Go, bringing p95 latency down to the old system's p50 level
• WARP plus Instant Connect let a client start a session with a single UDP packet
• An upcoming GPT-Live API is planned

If you build voice interfaces, or you're into latency optimization and real-time communication architecture, this one is worth a close read.