Slack replaced SSH-based job execution in its Amazon EMR data pipelines with a REST-driven orchestration layer called Quarry. The migration covered 700+ Airflow operators across eight data regions, eliminating direct SSH access to production clusters. Spark and Hive workloads were transitioned using Livy and HiveServer2, while arbitrary shell commands were handled via YARN's Distributed Shell capability. The three-quarter rollout was completed without downtime, improving security, reliability, and observability through server-side job lifecycle management, structured logging, and centralized metrics. Key challenges included virtual memory enforcement differences in YARN and cross-account network connectivity gaps uncovered during migration.
