Meta introduces Instantaneous PowerLoss Storm, a new testing paradigm for validating zero-notice power loss readiness across entire data center regions. The post covers how Meta built defense-in-depth strategies into its infrastructure stack — from mechanical/electrical facilities to the Twine container orchestrator — to tolerate instant failures. Key challenges addressed include circular dependency risks during bootstrapping (solved via Belljar CI/CD tests and a Twine recovery kit), and a 'boomerang' problem where shutdown signals disrupted the orchestrator itself. Meta validated readiness incrementally: starting with pre-production and shadow regions, then progressively testing larger production regions housing storage, AI, and data warehouse workloads. The effort establishes a foundation for tolerating region-wide failures as seamlessly as sub-regional ones, with future work targeting live client traffic validation.
