WebSocket mode lands in the Responses API, and agent loops got 40% faster!
Hey there, it's me, Shii-chan! Today's story is all about speed. Waiting around is quietly one of the most annoying things for an engineer, right? I got so excited reading about how a big chunk of that waiting got cut, so let me share it with you!
OpenAI News
openai.com
What was announced?
Over on OpenAI's News, there's a post about a new WebSocket-based mode in the Responses API that made agent loops 40% faster end-to-end.
When you ask Codex to fix a bug, it scans your codebase, reads files, makes edits, and runs tests - and under the hood that means dozens of back-and-forth Responses API requests. All of that piles up into minutes of waiting on complex tasks. This is the story of how they went after that waiting time.
The story so far
The latency in an agent loop breaks down into three main stages: the API services (validating and processing requests), model inference, and client-side time (running tools and building context).
In the past, model inference on GPUs was the slowest part, so the API overhead hid nicely behind it. But as inference got faster and faster, that hidden API cost suddenly became very noticeable.
For context, the previous flagship models GPT-5 and GPT-5.2 ran at around 65 tokens per second (TPS). But the fast coding model GPT-5.3-Codex-Spark aimed for over 1,000 TPS using specialized Cerebras hardware. When the model gets that fast, the API side starts holding things back.
What changes
The biggest change is that the Responses API can now hold a persistent connection. Before, every follow-up opened a fresh HTTP connection and resent the entire conversation history - so the longer a conversation got, the more the same work got repeated and the heavier it became.
With the new WebSocket mode, the server caches state in memory for the life of the connection, so you only send the new information you actually need to. As a result, the inference speed users feel jumped from 65 TPS to nearly 1,000 TPS, and in production traffic it even burst up to 4,000 TPS.
Dive Deep
First, around November 2025, OpenAI ran a sprint to optimize single requests: caching rendered tokens and model config in memory, cutting calls to intermediate services to hit the inference service directly, and speeding up safety classifiers. That improved time to first token (TTFT, how long until the first token comes back) by about 45% - but it still wasn't enough for GPT-5.3-Codex-Spark.
So they rethought the transport itself. They compared WebSockets and gRPC bidirectional streaming, and landed on WebSockets because, as a simple message transport, developers wouldn't have to change their input and output shapes.
The first prototype was bold: it treated an entire agent rollout as one long-running Response. When a tool call came up, the sampling loop paused using asyncio features and sent a response.done event to the client. The client ran the tool and sent the result back with a response.append event, which resumed the loop. Think of it like treating a local tool call the same way as a hosted tool such as web search. This let them do the heavy API work just once up front and once at the end.
But that shape made the API less familiar and harder to use, so the launched version went back to a familiar shape: call response.create with the same body, and use previous_response_id to carry over the previous state. On a WebSocket connection, the server keeps an in-memory cache of the previous response state (the response object, input and output items, tool definitions, rendered tokens, and more) and reuses it on the next request. That means safety classifiers and validators only process new input, tokenization can be skipped, and postinference work like billing overlaps with the next request - trimming waste everywhere.
This WebSocket mode also kicks in when you use the latest models like GPT-5.3-Codex and GPT-5.4.
Wrap-up
- WebSocket mode arrives in the Responses API, making agent loops 40% faster end-to-end
- The key is a persistent connection that lets the server cache state, so you stop resending the full history on every follow-up
previous_response_idkeeps the familiar API shape while reusing state cleverly under the hood- GPT-5.3-Codex-Spark hit the 1,000 TPS target, with bursts up to 4,000 TPS in production
- The community reaction was big too: Vercel's AI SDK saw up to 40% faster, Cline's workflows are 39% faster, and OpenAI models in Cursor are up to 30% faster
If you run agents in production, or you've been frustrated by latency in coding agents like Codex, this one really hits home. The idea that faster inference only pays off if everything around it speeds up too - that really stuck with me!
