--- title: The cluster that runs on our own metal: rebuilding... description: Propeller replaced a decade-old homegrown processing scheduler (3P) with a hybrid Kubernetes cluster spanning on-premise GPU servers in Sydney and Denver plus... canonical: https://daily.dev/posts/the-cluster-that-runs-on-our-own-metal-rebuilding-survey-processing-on-hybrid-kubernetes-4inmn4eto twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The cluster that runs on our own metal: rebuilding survey processing on hybrid Kubernetes | daily.dev og:description: Propeller replaced a decade-old homegrown processing scheduler (3P) with a hybrid Kubernetes cluster spanning on-premise GPU servers in Sydney and Denver plus... og:url: https://daily.dev/posts/the-cluster-that-runs-on-our-own-metal-rebuilding-survey-processing-on-hybrid-kubernetes-4inmn4eto og:image: https://api.daily.dev/og/posts/4inmn4EtO.png og:image:alt: The cluster that runs on our own metal: rebuilding survey processing on hybrid Kubernetes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The cluster that runs on our own metal: rebuilding survey processing on hybrid Kubernetes **[Medium](https://daily.dev/sources/medium_js)** · 11 min read · 0 upvotes · 0 comments ## Summary Propeller replaced a decade-old homegrown processing scheduler (3P) with a hybrid Kubernetes cluster spanning on-premise GPU servers in Sydney and Denver plus AWS EKS overflow. The control plane runs on EC2 while on-premise nodes join as workers over a site-to-site VPN. A custom Kubernetes operator handles cross-zone scheduling and storage via two custom resources (CrossZonePersistentVolumeClaim and CrossZoneJob), routing jobs to on-premise first and failing over to AWS only after time-based thresholds. Longhorn provides distributed block storage with ReadWriteMany support, enabling shared cache volumes across processing steps and live maintenance without downtime. Results include 100+ GPU nodes in production, 10,000+ concurrent pods at peak, 67,000+ daily processing steps, ~50% reduction in AlignSourcePhotos runtime, and AWS overflow costs of only ~$250/week versus tens of thousands for equivalent cloud GPUs. A second region in Denver was stood up on the same platform with no from-scratch build, proving the new architecture's scalability. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I build a Kubernetes cluster that spans both AWS and on-premise data centers? Run the control plane on EC2 in AWS for high availability, then join on-premise GPU and compute nodes as worker nodes over a site-to-site VPN so they form the data plane. A custom Kubernetes operator can then schedule jobs across on-premise zones first, failing over to AWS worker nodes only after time-based thresholds are exceeded, keeping cloud costs low while preserving on-premise cost advantages for GPU-heavy workloads. _Teams weighing hybrid Kubernetes tradeoffs can find more real-world infrastructure writeups like this on daily.dev._ ### How much does it cost to run GPU-heavy batch processing on-premise versus bursting to AWS? During peak weeks, overflowing GPU-heavy processing jobs into AWS cost only about $250 for the entire week, while the same workload on comparable cloud GPUs would cost tens of thousands of dollars weekly. Keeping the bulk of processing on owned on-premise hardware and reserving cloud bursting for genuine capacity overflow kept costs dramatically lower than a cloud-only GPU setup. _Engineers comparing on-premise versus cloud GPU costs can track similar cost-optimization case studies on daily.dev._ ### How does Longhorn's ReadWriteMany storage help with Kubernetes cluster maintenance? Longhorn keeps synchronized replicas of each cache volume across nodes, so a storage server can be pulled offline for servicing while jobs keep running off another replica without interrupting processing. Its ReadWriteMany mode also lets multiple pods read and write the same shared cache volume, letting a chain of processing steps work against one location instead of repeatedly pulling data from S3. _Developers managing distributed storage during maintenance windows can follow more Kubernetes storage patterns on daily.dev._ ## Similar posts on daily.dev - [𝐇𝐲𝐛𝐫𝐢𝐝 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭 𝐖𝐨𝐫𝐤𝐟𝐥𝐨𝐰: 𝐅𝐫𝐨𝐦 𝐎𝐧-𝐏𝐫𝐞𝐦 𝐭𝐨 𝐀𝐖𝐒](https://daily.dev/posts/35wh6slad) · Medium · 1 upvotes · 0 comments - [Multi-Cloud and Cross-Region GPU Capacity for Kubernetes AI](https://daily.dev/posts/multi-cloud-and-cross-region-gpu-capacity-for-kubernetes-ai-rzrutxnm5) · Cast AI · 0 upvotes · 0 comments - [Operating Kubernetes at scale: a few stories from running Amazon EKS](https://daily.dev/posts/operating-kubernetes-at-scale-a-few-stories-from-running-amazon-eks-vdcgnhxcm) · The New Stack · 1 upvotes · 0 comments - [Google Announces GKE Agent Sandbox and Hypercluster at Next '26, Positioning Kubernetes as AI Agent](https://daily.dev/posts/google-announces-gke-agent-sandbox-and-hypercluster-at-next-26-positioning-kubernetes-as-ai-agent-cfmrtqf9a) · InfoQ · 0 upvotes · 0 comments - [Amazon EKS now supports advanced Kubernetes control plane configuration parameters](https://daily.dev/posts/amazon-eks-now-supports-advanced-kubernetes-control-plane-configuration-parameters-prou1ktn3) · AWS · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#hybrid-cloud](https://daily.dev/tags/hybrid-cloud), [#gitops](https://daily.dev/tags/gitops) [View this post on daily.dev](https://daily.dev/posts/the-cluster-that-runs-on-our-own-metal-rebuilding-survey-processing-on-hybrid-kubernetes-4inmn4eto) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The cluster that runs on our own metal: rebuilding survey processing on hybrid Kubernetes","url":"https://daily.dev/posts/the-cluster-that-runs-on-our-own-metal-rebuilding-survey-processing-on-hybrid-kubernetes-4inmn4eto","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-cluster-that-runs-on-our-own-metal-rebuilding-survey-processing-on-hybrid-kubernetes-4inmn4eto"},"datePublished":"2026-07-25T07:32:45.818Z","dateModified":"2026-09-14T08:33:13.405Z","description":"Propeller replaced a decade-old homegrown processing scheduler (3P) with a hybrid Kubernetes cluster spanning on-premise GPU servers in Sydney and Denver plus...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0f4e14ca0239942464ff21c71f1a1210?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0f4e14ca0239942464ff21c71f1a1210?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-cluster-that-runs-on-our-own-metal-rebuilding-survey-processing-on-hybrid-kubernetes-4inmn4eto","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,hybrid-cloud,gitops","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"The cluster that runs on our own metal: rebuilding survey processing on hybrid Kubernetes"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-cluster-that-runs-on-our-own-metal-rebuilding-survey-processing-on-hybrid-kubernetes-4inmn4eto#faq","mainEntity":[{"@type":"Question","name":"How can I build a Kubernetes cluster that spans both AWS and on-premise data centers?","acceptedAnswer":{"@type":"Answer","text":"Run the control plane on EC2 in AWS for high availability, then join on-premise GPU and compute nodes as worker nodes over a site-to-site VPN so they form the data plane. A custom Kubernetes operator can then schedule jobs across on-premise zones first, failing over to AWS worker nodes only after time-based thresholds are exceeded, keeping cloud costs low while preserving on-premise cost advantages for GPU-heavy workloads. Teams weighing hybrid Kubernetes tradeoffs can find more real-world infrastructure writeups like this on daily.dev."}},{"@type":"Question","name":"How much does it cost to run GPU-heavy batch processing on-premise versus bursting to AWS?","acceptedAnswer":{"@type":"Answer","text":"During peak weeks, overflowing GPU-heavy processing jobs into AWS cost only about $250 for the entire week, while the same workload on comparable cloud GPUs would cost tens of thousands of dollars weekly. Keeping the bulk of processing on owned on-premise hardware and reserving cloud bursting for genuine capacity overflow kept costs dramatically lower than a cloud-only GPU setup. Engineers comparing on-premise versus cloud GPU costs can track similar cost-optimization case studies on daily.dev."}},{"@type":"Question","name":"How does Longhorn's ReadWriteMany storage help with Kubernetes cluster maintenance?","acceptedAnswer":{"@type":"Answer","text":"Longhorn keeps synchronized replicas of each cache volume across nodes, so a storage server can be pulled offline for servicing while jobs keep running off another replica without interrupting processing. Its ReadWriteMany mode also lets multiple pods read and write the same shared cache volume, letting a chain of processing steps work against one location instead of repeatedly pulling data from S3. Developers managing distributed storage during maintenance windows can follow more Kubernetes storage patterns on daily.dev."}}]} ```