--- title: Rerouting the Stream: How Lyft Moved to the Apache Flink... description: Lyft's Streaming Compute team migrated their entire streaming fleet from a homegrown Flink Kubernetes operator, built in 2020, to the open-source Apache Flink... canonical: https://daily.dev/posts/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-fbw5s7h5n twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Rerouting the Stream: How Lyft Moved to the Apache Flink Operator | daily.dev og:description: Lyft's Streaming Compute team migrated their entire streaming fleet from a homegrown Flink Kubernetes operator, built in 2020, to the open-source Apache Flink... og:url: https://daily.dev/posts/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-fbw5s7h5n og:image: https://api.daily.dev/og/posts/FbW5s7h5N.png og:image:alt: Rerouting the Stream: How Lyft Moved to the Apache Flink Operator og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Rerouting the Stream: How Lyft Moved to the Apache Flink Operator **[Lyft Engineering](https://daily.dev/sources/lyfteng)** · 14 min read · 0 upvotes · 0 comments ## Summary Lyft's Streaming Compute team migrated their entire streaming fleet from a homegrown Flink Kubernetes operator, built in 2020, to the open-source Apache Flink Kubernetes Operator. The move addressed mounting maintenance burden, a growing feature gap (autoscaling, autotuning, safe upgrades), and pinned outdated dependencies. The team built a translation layer converting their legacy FlinkApplication CRD into the Apache operator's FlinkDeployment CRD at the deploy-API boundary, allowing incremental, reversible migration without changing developer workflows. Follow-on work included adopting FlinkBlueGreenDeployments for zero-downtime deploys (including contributing an upstream bug fix), upgrading to Flink v1.19 for in-place autoscaling and Kinesis backlog metrics, moving Python SDK harness workloads to sidecars to fix autotuning OOM issues, and adopting Karpenter for dynamic node provisioning. The result: reduced operator maintenance, several million dollars per year in infrastructure savings, and Lyft becoming a contributor rather than sole maintainer of its streaming infrastructure. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why would a company replace a custom Flink Kubernetes operator with the open-source Apache Flink Kubernetes Operator? The main drivers are reduced maintenance burden, closing feature gaps, and avoiding dependency rot. Lyft's internally built operator from 2020 required custom code for every Flink upgrade, lacked autoscaling and automatic rollback, and was pinned to outdated libraries, while the Apache Flink Kubernetes Operator matured to offer intelligent autoscaling, automatic memory tuning, and safe upgrade mechanisms battle-tested at scale. _daily.dev surfaces engineering writeups like this for teams weighing a build-vs-adopt call on Kubernetes operators._ ### How can a team migrate custom Kubernetes CRDs to a new operator's CRD without breaking existing deployment workflows? Translate at the deploy API boundary rather than rewriting every application config. Lyft kept teams defining Flink jobs with their legacy FlinkApplication resource while a deploy API converted flat fields, upgrade modes, and pod templates into the Apache operator's nested FlinkDeployment spec, injecting legacy environment variables for backwards compatibility, and rolled out the change one application at a time starting in staging with instant rollback. _engineers planning a CRD migration can track these incremental rollout patterns on daily.dev._ ### What causes autotuning to OOM-kill TaskManagers when running Apache Beam Python SDK harness workloads on Flink? Autotuning reclaims JVM overhead memory it sees as unused by Flink, but Beam's Python SDK harness runs outside the Flink memory model and was relying on that same overhead allocation for years. Once autotuning began reclaiming those unused-looking pools, TaskManagers across the Beam fleet were OOM-killed; the fix was moving the Python harness out of the Flink container into its own sidecar with dedicated resource limits. _teams debugging Beam-on-Flink memory issues can find similar production postmortems on daily.dev._ ## Similar posts on daily.dev - [Apache Flink Kubernetes Operator 1.14.0 Release Announcement](https://daily.dev/posts/apache-flink-kubernetes-operator-1-14-0-release-announcement-rfruogk5n) · Flink · 3 upvotes · 0 comments - [A Tale of Two Flink Autoscalers](https://daily.dev/posts/a-tale-of-two-flink-autoscalers-cstvug6m2) · Netflix TechBlog · 7 upvotes · 1 comments - [LyftLearn Evolution: Rethinking ML Platform Architecture](https://daily.dev/posts/lyftlearn-evolution-rethinking-ml-platform-architecture-tpztmqapa) · Lyft Engineering · 0 upvotes · 0 comments - [Lyft Rearchitects ML Platform with Hybrid AWS SageMaker-Kubernetes Approach](https://daily.dev/posts/lyft-rearchitects-ml-platform-with-hybrid-aws-sagemaker-kubernetes-approach-ntubrebjb) · InfoQ · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#apache-flink](https://daily.dev/tags/apache-flink) [View this post on daily.dev](https://daily.dev/posts/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-fbw5s7h5n) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Rerouting the Stream: How Lyft Moved to the Apache Flink Operator","url":"https://daily.dev/posts/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-fbw5s7h5n","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-fbw5s7h5n"},"datePublished":"2026-08-31T19:20:49.756Z","dateModified":"2026-08-31T19:37:34.602Z","description":"Lyft's Streaming Compute team migrated their entire streaming fleet from a homegrown Flink Kubernetes operator, built in 2020, to the open-source Apache Flink...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/858ab9f47015b95ffa690014081809f9?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/858ab9f47015b95ffa690014081809f9?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Lyft Engineering","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Lyft Engineering","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/9573035416ea49d7a63042fbe1dd55d7","url":"https://daily.dev/sources/lyfteng"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-fbw5s7h5n","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,apache-flink","timeRequired":"PT14M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Lyft Engineering","item":"https://daily.dev/sources/lyfteng"},{"@type":"ListItem","position":3,"name":"Rerouting the Stream: How Lyft Moved to the Apache Flink Operator"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-fbw5s7h5n#faq","mainEntity":[{"@type":"Question","name":"Why would a company replace a custom Flink Kubernetes operator with the open-source Apache Flink Kubernetes Operator?","acceptedAnswer":{"@type":"Answer","text":"The main drivers are reduced maintenance burden, closing feature gaps, and avoiding dependency rot. Lyft's internally built operator from 2020 required custom code for every Flink upgrade, lacked autoscaling and automatic rollback, and was pinned to outdated libraries, while the Apache Flink Kubernetes Operator matured to offer intelligent autoscaling, automatic memory tuning, and safe upgrade mechanisms battle-tested at scale. daily.dev surfaces engineering writeups like this for teams weighing a build-vs-adopt call on Kubernetes operators."}},{"@type":"Question","name":"How can a team migrate custom Kubernetes CRDs to a new operator's CRD without breaking existing deployment workflows?","acceptedAnswer":{"@type":"Answer","text":"Translate at the deploy API boundary rather than rewriting every application config. Lyft kept teams defining Flink jobs with their legacy FlinkApplication resource while a deploy API converted flat fields, upgrade modes, and pod templates into the Apache operator's nested FlinkDeployment spec, injecting legacy environment variables for backwards compatibility, and rolled out the change one application at a time starting in staging with instant rollback. engineers planning a CRD migration can track these incremental rollout patterns on daily.dev."}},{"@type":"Question","name":"What causes autotuning to OOM-kill TaskManagers when running Apache Beam Python SDK harness workloads on Flink?","acceptedAnswer":{"@type":"Answer","text":"Autotuning reclaims JVM overhead memory it sees as unused by Flink, but Beam's Python SDK harness runs outside the Flink memory model and was relying on that same overhead allocation for years. Once autotuning began reclaiming those unused-looking pools, TaskManagers across the Beam fleet were OOM-killed; the fix was moving the Python harness out of the Flink container into its own sidecar with dedicated resource limits. teams debugging Beam-on-Flink memory issues can find similar production postmortems on daily.dev."}}]} ```