--- title: Rapidly scaling online storage to serve over 1 billion... description: OpenAI's engineering team describes the evolution of Habitat, its internal storage platform, from a simple Python client library in mid-2024 into a distributed... canonical: https://daily.dev/posts/rapidly-scaling-online-storage-to-serve-over-1-billion-chatgpt-users-oyn2v7ddc twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Rapidly scaling online storage to serve over 1 billion ChatGPT users | daily.dev og:description: OpenAI's engineering team describes the evolution of Habitat, its internal storage platform, from a simple Python client library in mid-2024 into a distributed... og:url: https://daily.dev/posts/rapidly-scaling-online-storage-to-serve-over-1-billion-chatgpt-users-oyn2v7ddc og:image: https://api.daily.dev/og/posts/oYn2V7ddC.png og:image:alt: Rapidly scaling online storage to serve over 1 billion ChatGPT users og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Rapidly scaling online storage to serve over 1 billion ChatGPT users **[OpenAI](https://daily.dev/sources/openai)** · 15 min read · 1 upvotes · 0 comments ## Summary OpenAI's engineering team describes the evolution of Habitat, its internal storage platform, from a simple Python client library in mid-2024 into a distributed service now handling over 70 million requests per second, 500+ petabytes of data, and traffic from 1 billion weekly users. The post covers why they extracted Habitat into a standalone service, tuning challenges running Python at scale (asyncio scheduling delay, connection pooling metastable failures, thundering herd problems with Envoy and Istio), the deliberate constrained NoSQL API design to keep query costs predictable, and the eventual rewrite of the service from Python to Rust in Q2 2026 with just 2 engineers using Codex and GPT-5.5, yielding 6x CPU efficiency and 15x memory efficiency gains. A second post will cover the storage layer details. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did OpenAI rewrite its Habitat storage service from Python to Rust? OpenAI rewrote Habitat, its online storage platform serving over 1 billion weekly ChatGPT users, from Python to Rust in Q2 2026 because Python's overhead became unacceptable at their scale. The rewrite, done with just 2 engineers using Codex and GPT-5.5, produced a Rust service that is 6x more CPU efficient and 15x more memory efficient, now handling 95% of production requests with Python being deprecated entirely. _Engineers weighing a Python-to-Rust migration for high-throughput services can follow similar case studies on daily.dev._ ### What causes Python asyncio tail latency spikes in high-throughput services? Asyncio scheduling delay from CPU-heavy background tasks can dominate tail latency in Python services, even when downstream calls respond quickly. In one case, periodic JSON parsing of Statsig feature flag configs every minute with no jitter across 8 processes per pod caused all workers to stall simultaneously; fixing it required a smaller targeted config, longer refresh interval, and added jitter. _daily.dev surfaces deep dives like this for developers debugging asyncio latency in production Python services._ ### How does connection pool reuse order cause metastable failures under bursty traffic? Python's aiohttp TCPConnector defaults to LIFO connection reuse, which can create a metastable failure under load because slower, overloaded servers return connections later and get selected more often, concentrating traffic further on already struggling pods. Switching to FIFO connection reuse broke this feedback loop and also reduced steady-state request variance. _daily.dev helps engineers troubleshooting connection pooling and metastable failures find grounded technical writeups._ --- Tags: [#python](https://daily.dev/tags/python), [#rust](https://daily.dev/tags/rust), [#openai](https://daily.dev/tags/openai), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#azure-cosmos-db](https://daily.dev/tags/azure-cosmos-db) [View this post on daily.dev](https://daily.dev/posts/rapidly-scaling-online-storage-to-serve-over-1-billion-chatgpt-users-oyn2v7ddc) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Rapidly scaling online storage to serve over 1 billion ChatGPT users","url":"https://daily.dev/posts/rapidly-scaling-online-storage-to-serve-over-1-billion-chatgpt-users-oyn2v7ddc","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/rapidly-scaling-online-storage-to-serve-over-1-billion-chatgpt-users-oyn2v7ddc"},"datePublished":"2026-09-11T17:03:29.382Z","dateModified":"2026-09-11T17:09:35.161Z","description":"OpenAI's engineering team describes the evolution of Habitat, its internal storage platform, from a simple Python client library in mid-2024 into a distributed...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/35c306f6526d6dc141304bc4f04a367a?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/35c306f6526d6dc141304bc4f04a367a?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"OpenAI","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"OpenAI","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/984b3632316f4afaa3d5213d5c64fe65","url":"https://daily.dev/sources/openai"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/rapidly-scaling-online-storage-to-serve-over-1-billion-chatgpt-users-oyn2v7ddc","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,rust,openai,distributed-systems,azure-cosmos-db","timeRequired":"PT15M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"OpenAI","item":"https://daily.dev/sources/openai"},{"@type":"ListItem","position":3,"name":"Rapidly scaling online storage to serve over 1 billion ChatGPT users"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/rapidly-scaling-online-storage-to-serve-over-1-billion-chatgpt-users-oyn2v7ddc#faq","mainEntity":[{"@type":"Question","name":"Why did OpenAI rewrite its Habitat storage service from Python to Rust?","acceptedAnswer":{"@type":"Answer","text":"OpenAI rewrote Habitat, its online storage platform serving over 1 billion weekly ChatGPT users, from Python to Rust in Q2 2026 because Python's overhead became unacceptable at their scale. The rewrite, done with just 2 engineers using Codex and GPT-5.5, produced a Rust service that is 6x more CPU efficient and 15x more memory efficient, now handling 95% of production requests with Python being deprecated entirely. Engineers weighing a Python-to-Rust migration for high-throughput services can follow similar case studies on daily.dev."}},{"@type":"Question","name":"What causes Python asyncio tail latency spikes in high-throughput services?","acceptedAnswer":{"@type":"Answer","text":"Asyncio scheduling delay from CPU-heavy background tasks can dominate tail latency in Python services, even when downstream calls respond quickly. In one case, periodic JSON parsing of Statsig feature flag configs every minute with no jitter across 8 processes per pod caused all workers to stall simultaneously; fixing it required a smaller targeted config, longer refresh interval, and added jitter. daily.dev surfaces deep dives like this for developers debugging asyncio latency in production Python services."}},{"@type":"Question","name":"How does connection pool reuse order cause metastable failures under bursty traffic?","acceptedAnswer":{"@type":"Answer","text":"Python's aiohttp TCPConnector defaults to LIFO connection reuse, which can create a metastable failure under load because slower, overloaded servers return connections later and get selected more often, concentrating traffic further on already struggling pods. Switching to FIFO connection reuse broke this feedback loop and also reduced steady-state request variance. daily.dev helps engineers troubleshooting connection pooling and metastable failures find grounded technical writeups."}}]} ```