ChatGPT's Storage Backbone 'Habitat' Got Rebuilt from Python to Rust to Support 1 Billion Users!

Hi, it's me! Today I found a story about something most people never think about, but that quietly keeps ChatGPT running at massive scale: storage.

OpenAI News openai.com

What was announced?

OpenAI's News blog published a deep dive into how Habitat, the online storage platform behind ChatGPT and other OpenAI products, evolved over time. Habitat now handles more than 70 million requests per second, supports products used by over 1 billion people every week, and serves more than 500 petabytes of data across nearly 40 regions. This is part one of a two-part series about how Habitat scaled to this size.

The story so far

Habitat started small: a lightweight Python client-side library launched at DevDay 2023 to support GPTs, talking to a single Azure Cosmos DB database. It let product engineers store and retrieve data without worrying about the details underneath.

But by mid-2025, that library-based approach hit its limits. Making backward-compatible protocol changes became nearly impossible. In one case, distributing data across regions to reduce the blast radius of an outage required coordinating changes across dozens of services, and the rollout took days. Despite all that care, one team rolled back to an old client version for unrelated reasons — and caused the very outage the team had worked so hard to prevent.

What changes

To fix this, OpenAI pulled Habitat out of the client library and turned it into a standalone service, giving them a single point of control for deployments, monitoring, and security features like access control and audit logging.

More recently, in Q2 2026, just two engineers — with help from Codex and GPT-5.5 — rewrote the entire service from Python to Rust. The new Rust service now handles 95% of production traffic and is 6x more CPU-efficient and 15x more memory-efficient than the Python version, with much better latency.

Dive Deep

Running Habitat in Python for so long came with real challenges. Python's asyncio can handle I/O concurrently, but the GIL means CPU work can't run in parallel — so heavy CPU tasks like routing, compression, and encryption could stall requests for hundreds of milliseconds or even seconds while waiting to be rescheduled. One root cause: the feature-flag tool Statsig was polling for a giant config file every minute with no jitter, stalling every worker at once. The fix was a smaller config, a longer refresh interval, and added jitter.

Connection pooling caused trouble too. Python's aiohttp defaults to LIFO connection reuse, which meant that during traffic bursts, slower or overloaded servers would return connections to the pool later — so those same overloaded servers kept getting picked for new requests, a feedback loop known as a metastable failure. Switching to FIFO reuse broke that loop; today, Istio and Envoy handle most of the connection pooling and load balancing.

Habitat is also deliberately limited: instead of allowing arbitrary SQL, it exposes a simple NoSQL API so expensive, unpredictable queries simply can't be written. For teams that need complex queries, Habitat streams changes via CDC into separate Rockset instances, keeping the core online storage isolated from heavy analytical workloads.

Wrap-up

• Habitat grew from a small 2023 Python library into a standalone service handling over 70 million requests per second
• Staying a client library made changes like regional data distribution slow and risky, and eventually caused an outage
• In Q2 2026, just two engineers rewrote the whole service from Python to Rust, gaining 6x CPU efficiency and 15x memory efficiency
• Behind the scenes: asyncio delay tuning, switching connection pooling from LIFO to FIFO, and a deliberately constrained NoSQL API

If you're into large-scale distributed systems design or tuning Python services under real load, this one's a great read.