12x Faster ChatGPT Reads! Inside OpenAI's Habitat and Cosmos DB Journey

Hi everyone, it's Shiichan!

OpenAI News openai.com

Today I read Part II of OpenAI's series on scaling its storage platform. The story behind serving over 1 billion users was a great read!

What was announced?

OpenAI News published Part II of its series on scaling the Habitat storage platform. Part I covered Habitat's evolution from a Python client into a service. This part focuses on how OpenAI scaled the underlying store, Azure Cosmos DB.

Habitat is the online storage platform that lets OpenAI products access the information they need quickly and reliably. It has to respond in milliseconds to over 1 billion users across 40 geographic regions, and it serves over 500 petabytes of data.

Why it matters

This post is a real-world look at running a huge multi-tenant database and working with a managed service at extreme scale. OpenAI's philosophy is reliability first, then performance, then efficiency, and the article shows what was fixed in that order.

What changes

The headline result is read latency. Global read p50 dropped from roughly 120 ms to roughly 10 ms, a 12x improvement. Reads make up over 90% of Habitat's requests, so this matters a lot.

The goal is also that product developers no longer need client-side caching and batching tricks just to hide slow reads.

Dive Deep

The multi-tenancy journey

OpenAI went around in a circle: shared container, then dedicated containers, then back to a shared container.

In Phase I (one shared container), problems included:

• Logical partition keys such as user_id and workspace_id collided across use cases, limiting per-user throughput
• Limits are really enforced per physical partition, so co-located keys caused noisy neighbors, where niche features could disrupt login and signup
• When data grew and physical partitions split, provisioned throughput was divided among more partitions, so the per-partition limit shrank (weak partitions)

Phase II (dedicated containers) gave isolation, but a feature jumping from 0 to 1 million QPS needed pre-provisioning. Teams forgot to pre-scale, launch days turned into last-minute scaling, and nearly every product team ended up involved in capacity planning.

Phase III returned to a shared container, with these fixes built together with the Azure team:

• Hot key throttling: throttle the hottest keys first to limit the blast radius
• Quality of Service (QoS): throttle offline work such as bulk inserts and extracts before interactive requests
• A new serverless mode: each physical partition scales up to 10K RU/s, with splits based on RU usage as well as storage

This removed the need for manual capacity planning, and the article says there have been no provisioning or noisy-neighbor incidents in the months since.

Co-managed operations

The largest Azure Cosmos DB account stores tens of petabytes and serves more than 20M QPS. A regional outage in late 2024 showed that failing over the largest accounts took several hours. About a month later, failing back caused a connection stampede that overwhelmed the original region's gateway.

In response, the two teams moved to shared tests, shared dashboards, and direct engineering channels. They test changes on a production-sized non-production account, iterate on SDKs together, improved connection pooling and reuse, and isolated critical workloads from offline ones using different VPCs.

Global performance

The read strategy has layers:

• Store data near users when capacity allows (limited by data center capacity, power, and cooling)
• Broadly replicate only the smallest, most latency-sensitive datasets, such as authentication metadata, in some cases to all 40 regions
• Rely on integrated caching for everything else

The cache is a lookaside layer on Valkey servers. To avoid the TTL trade-off, invalidations follow a TAO-like approach and are propagated globally within seconds from the Azure Cosmos DB change feed (CDC). Rewriting the Habitat service from Python to Rust cut more overhead.

CDC itself carries millions of changes per second through more than a hundred Kafka topics, also powering secondary indexes, data movement, and offline analytics.

Wrap-up

• Habitat supports over 1 billion users in 40 regions with over 500 PB of data
• Multi-tenancy returned to a shared container with hot key throttling, QoS, and a new serverless mode
• At this scale, managed services need co-management with the provider
• Valkey caching, CDC-based invalidation, and Rust cut read p50 from about 120 ms to about 10 ms

If you design or operate large data platforms, this is a rich read for you!