---
title: "From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines"
url: https://daily.dev/posts/from-ssh-to-rest-a-security-driven-modernization-of-slack-s-emr-data-pipelines-idqy8rey4
source_url: https://slack.engineering/from-ssh-to-rest-a-security-driven-modernization-of-slacks-emr-data-pipelines
type: article
source: "Slack engineering"
published: 2026-05-05T14:03:57.786Z
updated: 2026-05-05T14:04:43.370Z
tags: ["aws", "devops", "yarn", "apache-airflow"]
reading_time: 15
upvotes: 61
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines

**[Slack engineering](https://daily.dev/sources/slack)** · 15 min read · 61 upvotes · 0 comments

## Summary

Slack's data platform had accumulated 700+ SSH-based Airflow operators running jobs on AWS EMR clusters by 2024, creating security risks, operational fragility, and blocking infrastructure modernization. The team migrated all jobs to a REST-based architecture using Quarry, an internal job submission gateway, and YARN's Distributed Shell feature to handle arbitrary shell commands without SSH. The migration covered 8 data regions, 7 operator types, and 5 teams over 3 quarters with zero downtime. Key challenges included YARN virtual memory check failures (previously hidden by SSH), network topology mismatches across account boundaries, and multi-region coordination complexity. Results included elimination of SSH attack surface, improved job reliability (jobs survive Kubernetes pod restarts), better observability, and unblocking of Spark-on-Kubernetes migration.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://slack.engineering/from-ssh-to-rest-a-security-driven-modernization-of-slacks-emr-data-pipelines>

## Similar posts on daily.dev

- [Slack Eliminates SSH in EMR Pipelines, Migrates 700\+ Jobs to Rest-Based Architecture](https://daily.dev/posts/slack-eliminates-ssh-in-emr-pipelines-migrates-700-jobs-to-rest-based-architecture-noyvsryqz) · InfoQ · 1 upvotes · 0 comments

---

Tags: [#aws](https://daily.dev/tags/aws), [#devops](https://daily.dev/tags/devops), [#yarn](https://daily.dev/tags/yarn), [#apache-airflow](https://daily.dev/tags/apache-airflow)

[View this post on daily.dev](https://daily.dev/posts/from-ssh-to-rest-a-security-driven-modernization-of-slack-s-emr-data-pipelines-idqy8rey4)