Meta's engineering team details how they rebuilt their BLOB-storage architecture to meet the demands of large-scale AI training workloads. The legacy system, built for traditional web apps, suffered from multi-layer metadata lookups causing hundreds of milliseconds of latency — unacceptable for GPU-bound training jobs. The new architecture collapses metadata into a unified flat schema backed by ZippyDB for O(1) lookups, eliminates the dataplane proxy in favor of a fat client SDK that streams directly from Tectonic storage nodes, and deploys regional BLOB-storage stacks colocated with GPUs. To handle hot spots and traffic spikes, Meta leverages spare GPU host memory as a distributed data cache (achieving ~80% hit rates) and caches read-plan metadata for 1-2ms access. For research velocity, they introduced a tiered caching model analogous to OS page caching: GPU host memory and flash serve as L1/L2 caches, regional disaggregated flash as L3, and global HDD-backed BLOB storage as the source of truth. Explicit prefetch APIs and automatic data lifecycle management allow researchers to skip manual cross-region data ingestion, reducing iteration time from hours to minutes.
