--- title: Kubernetes on Oxide: How Customer Needs Shaped Our... description: Oxide Computer describes how it built a set of Kubernetes integrations driven directly by customer needs, covering the full cluster lifecycle. Provisioning is... canonical: https://daily.dev/posts/kubernetes-on-oxide-how-customer-needs-shaped-our-integrations-vdr4ohsdf twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Kubernetes on Oxide: How Customer Needs Shaped Our Integrations | daily.dev og:description: Oxide Computer describes how it built a set of Kubernetes integrations driven directly by customer needs, covering the full cluster lifecycle. Provisioning is... og:url: https://daily.dev/posts/kubernetes-on-oxide-how-customer-needs-shaped-our-integrations-vdr4ohsdf og:image: https://api.daily.dev/og/posts/vdr4OHsDf.png og:image:alt: Kubernetes on Oxide: How Customer Needs Shaped Our Integrations og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Kubernetes on Oxide: How Customer Needs Shaped Our Integrations **[Oxide](https://daily.dev/sources/oxide)** · 13 min read · 23 upvotes · 1 comments ## Summary Oxide Computer describes how it built a set of Kubernetes integrations driven directly by customer needs, covering the full cluster lifecycle. Provisioning is handled through three paths: a Rancher node driver, an Omni infrastructure provider for Talos Linux (built in partnership with Sidero Labs), and a Cluster API provider (CAPOx). A cloud controller manager reconciles Oxide instances with Kubernetes Node objects and implements LoadBalancer services using Oxide's floating IPs, publishing both a floating IP and a node internal IP in the service status. For storage, Oxide currently recommends Longhorn backed by newly introduced Oxide local disks to avoid double replication, while a native CSI plugin is blocked on adding disk hot-plug support across the Oxide stack. The post closes with a roadmap covering CSI completion, autoscaling, external subnet support, and future native load balancing. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does Oxide support Kubernetes LoadBalancer services without a native load balancer? Oxide's cloud controller manager uses floating IPs to back LoadBalancer services. A floating IP delivers traffic to a single Kubernetes node, and the Kubernetes Service dataplane distributes it to the right pods; the service controller publishes both the floating IP (Proxy mode) and the node's internal IP (VIP mode) in status.loadBalancer.ingress since Oxide translates the destination address before it reaches the guest. _Teams weighing load-balancer workarounds on bare-metal platforms can track integration details like this on daily.dev._ ### Why hasn't Oxide released a native CSI plugin for Kubernetes persistent volumes yet? Oxide requires an instance to be stopped before a disk can be attached or detached, but Kubernetes expects a CSI driver to attach storage to a running worker after scheduling a pod. Stopping the worker would disrupt every other workload on that node, so Oxide must add disk hot-plug support across its hypervisor and API stack before releasing its native CSI plugin. _Anyone planning stateful workloads on Oxide can follow storage integration progress like this via daily.dev._ ### How can I avoid double replication when running Longhorn on Oxide distributed disks? Back Longhorn's replicas with Oxide local disks instead of Oxide distributed disks. Distributed disks already store three replicas across distinct sleds, so layering a three-replica Longhorn volume on top can fan out one application write to as many as nine physical disk writes; local disks have no built-in replication, avoiding that duplication. _Engineers tuning storage architecture on Oxide can keep up with configuration details like this through daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 43 comments across hackernews (as of 2026-09-14). **TL;DR:** Discussion centers on curiosity about where Oxide fits versus baremetal Talos/k8s or CoreWeave-style clouds, with Oxide engineers actively engaging on storage, CCM/CAPI/Karpenter roadmap details, and a side debate about home-lab demand versus enterprise focus. **Sentiment:** 45% positive · 40% mixed · 15% skeptical **The case for** - Engineers from Oxide are actively and substantively answering technical questions in-thread, which people appreciate. - The out-of-tree CCM and CAPI provider approach is seen as a good architectural choice with room to differentiate (e.g. Karpenter fit). - Some praised the willingness to open source their RFD/documentation system. - Eliminating the network overlay via external subnets is viewed as a genuine advantage over public clouds. **The pushback** - Several people question why Longhorn was chosen over alternatives like Mayastor given feature gaps (e.g. NVMe-oF). - The per-PVC volume attach model seems awkward, with concerns about RWX PVC support and whether NFS would be better for some workloads. - Kubernetes on Oxide is seen as not yet matching public cloud (e.g. no per-pod VM isolation like Fargate, incomplete native integration). - One commenter argues Oxide's engineering/marketing focus is misdirected away from a nonexistent home-lab market, though this view was pushed back on. - A UI bug report about the site's top navigation being unusable on iPad/Apple Pencil. **By community** - hackernews (mixed): Genuine technical curiosity and roadmap discussion coexist with pointed critiques of storage choices and a heated tangent about home-lab versus enterprise market focus. **Hottest debate:** Whether Oxide's enterprise-rack focus is the right strategy versus the vocal home-lab enthusiasm seen in comments, with Oxide's own team pushing back on the criticism. **Open questions** - Why Longhorn was chosen over alternatives like Mayastor that have more mature NVMe-oF support. - How RWX PVCs and multi-node volume mounting will be handled without hot-plug support. - What the boundary will look like between Cluster API-based node management and a future Karpenter-based autoscaler. - Whether path-based provisioning on a single large volume could work as an alternative to per-PVC volume attachment. **Highlights** > (I think you meant "like hotcakes", not "like cupcakes"?) In any event: from the beginning, we have always been targeted at the enterprise buyer who is looking at annual public cloud bills in the tens, hundreds or (in some cases) thousands of millions of dollars per year. We love the enthusiasm that Oxide engenders among the home lab set, but that's not how we have geared the business (for lots of good reasons). Also, for whatever it's worth: people do want to buy them, it turns out. > — [bcantrill on hackernews](https://news.ycombinator.com/item?id=49290281) > why longhorn and not mayastor? longhorn doesn't even have nvmeof that is not experimental > — [dilyevsky on hackernews](https://news.ycombinator.com/item?id=49290147) > So you’d be attaching a new volume to the running worker VM for each PVC? This seems a little odd to me. Could you attach a single large volume, and do path-based provisioning on that? > — [wolttam on hackernews · 2 comments](https://news.ycombinator.com/item?id=49288521) > Every single Oxide article has comments about people wanting one at home. A great example of completely misdirected marketing and/or engineering. Oxide should have developed home microcomputers, and they would have sold like cupcakes, with their ASCII art marketing. Instead they do million dollar "mainframes" that no one (except VCs) wants to buy. > — [lkasjdas on hackernews · 3 comments](https://news.ycombinator.com/item?id=49290161) > Karpenter is definitely a natural fit for Oxide. There's some interesting boundaries that we're discussing internally, like @sudomateo mentioned. One of which is CAPI. There's a CAPI Karpenter provider that elmiko built (notably back in the very early days of Karpenter). I think there's room for both use-cases. Some may want to use CAPI for everything and others may want a platform that is based on CAPI but guest clusters are not. One implementation specific detail which makes Karpenter interesting on Oxide is the tunable CPU and memory parameters rather than strict instance type shapes. The prototype I built for Karpenter on Oxide generates all possible "instance type" combinations, so you can create some really specific nodes to bin-pack pods. Another interesting area is multiple providers. This is becoming pretty common across public clouds too. A multi-provider Karpenter is something I'm interested in and I know some folks have already been gluing together, but the Karpenter story isn't great on maintaining those since you basically need to compile them together today. CAPIs multi-provider story is a bit cleaner since it only relies on CRDs. If you have ideas, let's chat in the Kubernetes slack #karpenter-dev. > — [bmwagner10 on hackernews](https://news.ycombinator.com/item?id=49288949) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49286485) · 95 points · 43 comments ## Community discussion Top comments from developers on daily.dev. **@agustinbarrientos** · 1 upvotes > What happens to a long-lived connection while the floating IP moves to another node? ## Similar posts on daily.dev - [Building a cloud native platform from the ground up with Kairos, k0rdent, and bindy](https://daily.dev/posts/building-a-cloud-native-platform-from-the-ground-up-with-kairos-k0rdent-and-bindy-77dhrzmko) · CNCF · 7 upvotes · 0 comments - [The cluster that runs on our own metal: rebuilding survey processing on hybrid Kubernetes](https://daily.dev/posts/the-cluster-that-runs-on-our-own-metal-rebuilding-survey-processing-on-hybrid-kubernetes-4inmn4eto) · Medium · 0 upvotes · 0 comments - [Hosting and scaling EKS hybrid nodes with KubeVirt and Kube-OVN CNI](https://daily.dev/posts/hosting-and-scaling-eks-hybrid-nodes-with-kubevirt-and-kube-ovn-cni-snalwzpbd) · ITNEXT · 0 upvotes · 0 comments - [𝐇𝐲𝐛𝐫𝐢𝐝 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭 𝐖𝐨𝐫𝐤𝐟𝐥𝐨𝐰: 𝐅𝐫𝐨𝐦 𝐎𝐧-𝐏𝐫𝐞𝐦 𝐭𝐨 𝐀𝐖𝐒](https://daily.dev/posts/35wh6slad) · Medium · 1 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes) [View this post on daily.dev](https://daily.dev/posts/kubernetes-on-oxide-how-customer-needs-shaped-our-integrations-vdr4ohsdf) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Kubernetes on Oxide: How Customer Needs Shaped Our Integrations","url":"https://daily.dev/posts/kubernetes-on-oxide-how-customer-needs-shaped-our-integrations-vdr4ohsdf","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/kubernetes-on-oxide-how-customer-needs-shaped-our-integrations-vdr4ohsdf"},"datePublished":"2026-08-13T09:08:44.215Z","dateModified":"2026-09-14T06:54:11.658Z","description":"Oxide Computer describes how it built a set of Kubernetes integrations driven directly by customer needs, covering the full cluster lifecycle. Provisioning is...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a6c69f26084724534d5eb6a0ae337988?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a6c69f26084724534d5eb6a0ae337988?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Oxide","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Oxide","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/a06afd0e0fbf40ec9d3128cd1c103021","url":"https://daily.dev/sources/oxide"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/kubernetes-on-oxide-how-customer-needs-shaped-our-integrations-vdr4ohsdf","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":23},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"kubernetes","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Oxide","item":"https://daily.dev/sources/oxide"},{"@type":"ListItem","position":3,"name":"Kubernetes on Oxide: How Customer Needs Shaped Our Integrations"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/kubernetes-on-oxide-how-customer-needs-shaped-our-integrations-vdr4ohsdf","comment":[{"@type":"Comment","text":"What happens to a long-lived connection while the floating IP moves to another node?","datePublished":"2026-08-15T21:57:10.696Z","url":"https://daily.dev/posts/vdr4OHsDf#c-W9hQnoHZp","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/kubernetes-on-oxide-how-customer-needs-shaped-our-integrations-vdr4ohsdf#faq","mainEntity":[{"@type":"Question","name":"How does Oxide support Kubernetes LoadBalancer services without a native load balancer?","acceptedAnswer":{"@type":"Answer","text":"Oxide's cloud controller manager uses floating IPs to back LoadBalancer services. A floating IP delivers traffic to a single Kubernetes node, and the Kubernetes Service dataplane distributes it to the right pods; the service controller publishes both the floating IP (Proxy mode) and the node's internal IP (VIP mode) in status.loadBalancer.ingress since Oxide translates the destination address before it reaches the guest. Teams weighing load-balancer workarounds on bare-metal platforms can track integration details like this on daily.dev."}},{"@type":"Question","name":"Why hasn't Oxide released a native CSI plugin for Kubernetes persistent volumes yet?","acceptedAnswer":{"@type":"Answer","text":"Oxide requires an instance to be stopped before a disk can be attached or detached, but Kubernetes expects a CSI driver to attach storage to a running worker after scheduling a pod. Stopping the worker would disrupt every other workload on that node, so Oxide must add disk hot-plug support across its hypervisor and API stack before releasing its native CSI plugin. Anyone planning stateful workloads on Oxide can follow storage integration progress like this via daily.dev."}},{"@type":"Question","name":"How can I avoid double replication when running Longhorn on Oxide distributed disks?","acceptedAnswer":{"@type":"Answer","text":"Back Longhorn's replicas with Oxide local disks instead of Oxide distributed disks. Distributed disks already store three replicas across distinct sleds, so layering a three-replica Longhorn volume on top can fan out one application write to as many as nine physical disk writes; local disks have no built-in replication, avoiding that duplication. Engineers tuning storage architecture on Oxide can keep up with configuration details like this through daily.dev."}}]} ```