--- title: From SSH to REST: A Security-Driven Modernization of... description: Slack's data platform had accumulated 700+ SSH-based Airflow operators running jobs on AWS EMR clusters by 2024, creating security risks, operational... canonical: https://daily.dev/posts/from-ssh-to-rest-a-security-driven-modernization-of-slack-s-emr-data-pipelines-idqy8rey4 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines | daily.dev og:description: Slack's data platform had accumulated 700+ SSH-based Airflow operators running jobs on AWS EMR clusters by 2024, creating security risks, operational... og:url: https://daily.dev/posts/from-ssh-to-rest-a-security-driven-modernization-of-slack-s-emr-data-pipelines-idqy8rey4 og:image: https://api.daily.dev/og/posts/IDQy8ReY4.png og:image:alt: From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines **[Slack engineering](https://daily.dev/sources/slack)** · 15 min read · 61 upvotes · 0 comments ## Summary Slack's data platform had accumulated 700+ SSH-based Airflow operators running jobs on AWS EMR clusters by 2024, creating security risks, operational fragility, and blocking infrastructure modernization. The team migrated all jobs to a REST-based architecture using Quarry, an internal job submission gateway, and YARN's Distributed Shell feature to handle arbitrary shell commands without SSH. The migration covered 8 data regions, 7 operator types, and 5 teams over 3 quarters with zero downtime. Key challenges included YARN virtual memory check failures (previously hidden by SSH), network topology mismatches across account boundaries, and multi-region coordination complexity. Results included elimination of SSH attack surface, improved job reliability (jobs survive Kubernetes pod restarts), better observability, and unblocking of Spark-on-Kubernetes migration. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Slack Eliminates SSH in EMR Pipelines, Migrates 700\+ Jobs to Rest-Based Architecture](https://daily.dev/posts/slack-eliminates-ssh-in-emr-pipelines-migrates-700-jobs-to-rest-based-architecture-noyvsryqz) · InfoQ · 1 upvotes · 0 comments - [EverC: Regaining Control of Data Platform Costs on AWS with EMR on EKS](https://daily.dev/posts/everc-regaining-control-of-data-platform-costs-on-aws-with-emr-on-eks-x1fldlsgc) · BigData Boutique blog · 1 upvotes · 0 comments - [Slack AI: The Path to Multi-Cloud](https://daily.dev/posts/slack-ai-the-path-to-multi-cloud-mcgkrjcli) · Slack engineering · 0 upvotes · 0 comments - [Serverless Storage on Amazon EMR Serverless now supports terabyte-scale shuffle](https://daily.dev/posts/serverless-storage-on-amazon-emr-serverless-now-supports-terabyte-scale-shuffle-f9e5ihrmn) · AWS · 0 upvotes · 0 comments - [Migrating Your Shopify Data Pipeline from REST to GraphQL with Openflow and Snowflake](https://daily.dev/posts/migrating-your-shopify-data-pipeline-from-rest-to-graphql-with-openflow-and-snowflake-7e3yw1frz) · Snowflake Community · 0 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#devops](https://daily.dev/tags/devops), [#yarn](https://daily.dev/tags/yarn), [#apache-airflow](https://daily.dev/tags/apache-airflow) [View this post on daily.dev](https://daily.dev/posts/from-ssh-to-rest-a-security-driven-modernization-of-slack-s-emr-data-pipelines-idqy8rey4) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines","url":"https://daily.dev/posts/from-ssh-to-rest-a-security-driven-modernization-of-slack-s-emr-data-pipelines-idqy8rey4","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/from-ssh-to-rest-a-security-driven-modernization-of-slack-s-emr-data-pipelines-idqy8rey4"},"datePublished":"2026-05-05T14:03:57.786Z","dateModified":"2026-05-05T14:04:43.370Z","description":"Slack's data platform had accumulated 700+ SSH-based Airflow operators running jobs on AWS EMR clusters by 2024, creating security risks, operational...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9d3e72a547bb8434c91cf72800348388?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9d3e72a547bb8434c91cf72800348388?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Slack engineering","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Slack engineering","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/f96745b95f204498816ad834bf964380","url":"https://daily.dev/sources/slack"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/from-ssh-to-rest-a-security-driven-modernization-of-slack-s-emr-data-pipelines-idqy8rey4","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":61},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,devops,yarn,apache-airflow","timeRequired":"PT15M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Slack engineering","item":"https://daily.dev/sources/slack"},{"@type":"ListItem","position":3,"name":"From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines"}]} ```