Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

OpenAI's Habitat is its online storage platform serving over 1 billion users across 40 regions, storing 500 petabytes of data. Part II of this series details how OpenAI scaled its use of Azure Cosmos DB through a close partnership with Microsoft's Azure team. The article traces three phases: Phase I relied on Azure Cosmos DB primitives like logical partition keys, which led to throughput collisions on hot keys, noisy-neighbor interference across physical partitions, and shrinking per-partition capacity as storage grew. Phase II tried workload isolation with single-tenant containers, which solved isolation problems but created provisioning overhead, unstable aggregate growth, and high capacity-planning burden on product teams. Phase III returned to multi-tenancy by default, using a co-managed container design with new Azure Cosmos DB features: hot key throttling, quality-of-service request tiering, and an improved serverless mode with RU-usage-based splits. The article also discusses co-engineering with Azure (shared dashboards, tests, and joint incident response) and a significant 2024 regional outage that revealed untested failover behavior at OpenAI's scale. Finally, it covers performance work: multi-region replication for latency-critical datasets, an integrated distributed cache using change data capture (CDC) for invalidation, and a Rust rewrite of the Habitat service that cut global read p50 latency from ~120 ms to ~10 ms. Looking ahead, OpenAI is building a custom in-house exabyte-scale storage system combining Rockset's RocksDB Cloud with Raft-based replication to address future replication challenges.

(Source：OpenAI)