--- title: Eleven minutes, zero humans: Building a self-healing... description: A homelab-scale walkthrough describes building a fully automated, self-healing Kubernetes control plane upgrade pipeline using Kairos Hadron (an immutable OS... canonical: https://daily.dev/posts/eleven-minutes-zero-humans-building-a-self-healing-kubernetes-upgrade-pipeline-on-kairos-lzagvtph7 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Eleven minutes, zero humans: Building a self-healing Kubernetes upgrade pipeline on Kairos | daily.dev og:description: A homelab-scale walkthrough describes building a fully automated, self-healing Kubernetes control plane upgrade pipeline using Kairos Hadron (an immutable OS... og:url: https://daily.dev/posts/eleven-minutes-zero-humans-building-a-self-healing-kubernetes-upgrade-pipeline-on-kairos-lzagvtph7 og:image: https://api.daily.dev/og/posts/lzagvtPh7.png og:image:alt: Eleven minutes, zero humans: Building a self-healing Kubernetes upgrade pipeline on Kairos og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Eleven minutes, zero humans: Building a self-healing Kubernetes upgrade pipeline on Kairos **[CNCF](https://daily.dev/sources/cncf)** · 5 min read · 8 upvotes · 0 comments ## Summary A homelab-scale walkthrough describes building a fully automated, self-healing Kubernetes control plane upgrade pipeline using Kairos Hadron (an immutable OS with A/B partition upgrades), k3s, Renovate, Kyverno, Cosign, ArgoCD, and kairos-operator. A real upgrade run from v0.3.0 to v0.4.0 across three control plane nodes completed in 11 minutes with zero human intervention and no etcd quorum loss. The piece also details a subtle bug: a Renovate custom regex field (extractVersionTemplate) that silently failed to bump the CR name, requiring a switch to currentValueTemplate to fix version comparison logic. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I automatically trigger a Kairos NodeOpUpgrade custom resource when a new image is published? Renovate must bump metadata.name in the same commit as spec.image, not just the image tag alone. NodeOpUpgrade is a one-shot resource that kairos-operator marks complete and never reprocesses, so patching spec.image on an existing CR does nothing; changing the name forces ArgoCD to delete the old CR and create a new one, which is the actual trigger for the upgrade. _daily.dev surfaces GitOps and Kubernetes operator gotchas like this for engineers automating cluster upgrades._ ### What does concurrency: 0 mean in a Kairos upgrade spec, and is it safe for a multi-node control plane? Concurrency: 0 in a Kairos upgrade spec means all nodes upgrade simultaneously, not one at a time as might be assumed. This caused three control-plane nodes to reboot at once during a homelab test; etcd quorum survived only by luck. Setting concurrency: 1 forces a proper rolling, one-node-at-a-time upgrade sequence. _Teams rolling out Kairos or similar immutable-OS upgrades can track configuration pitfalls like this on daily.dev._ ### Why did my Renovate custom regex manager silently fail to update a version string formatted with dashes? The extractVersionTemplate field does not exist in Renovate's custom manager schema, so using it silently does nothing rather than throwing an error. Switching to currentValueTemplate correctly converts a dash-formatted version like v0-3-0 into semver v0.3.0 for comparison, then reconstructs the dash format for the new tag, fixing the stalled bump. _daily.dev helps developers debugging silent Renovate config failures find fixes like this one faster._ ## Similar posts on daily.dev - [How We Cut Kubernetes Deployment Validation From 45 Minutes to 2 minutes](https://daily.dev/posts/how-we-cut-kubernetes-deployment-validation-from-45-minutes-to-2-minutes-f8ygqwvdh) · Cloud Native Now · 24 upvotes · 2 comments - [A live Kubernetes cluster can still have an ownership gap](https://daily.dev/posts/a-live-kubernetes-cluster-can-still-have-an-ownership-gap-by8o4s5rz) · The New Stack · 1 upvotes · 0 comments - [Kubernetes gets minor version rollback](https://daily.dev/posts/kubernetes-gets-minor-version-rollback-qfydcuwrs) · Google Cloud · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#cicd](https://daily.dev/tags/cicd), [#gitops](https://daily.dev/tags/gitops), [#argocd](https://daily.dev/tags/argocd) [View this post on daily.dev](https://daily.dev/posts/eleven-minutes-zero-humans-building-a-self-healing-kubernetes-upgrade-pipeline-on-kairos-lzagvtph7) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Eleven minutes, zero humans: Building a self-healing Kubernetes upgrade pipeline on Kairos","url":"https://daily.dev/posts/eleven-minutes-zero-humans-building-a-self-healing-kubernetes-upgrade-pipeline-on-kairos-lzagvtph7","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/eleven-minutes-zero-humans-building-a-self-healing-kubernetes-upgrade-pipeline-on-kairos-lzagvtph7"},"datePublished":"2026-08-14T11:16:42.122Z","dateModified":"2026-09-14T06:17:19.495Z","description":"A homelab-scale walkthrough describes building a fully automated, self-healing Kubernetes control plane upgrade pipeline using Kairos Hadron (an immutable OS...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/825f524ec07abf4a691dd8885b99ccdd?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/825f524ec07abf4a691dd8885b99ccdd?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"CNCF","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"CNCF","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/55536a03390741eebd96d2f9b74d2d8d","url":"https://daily.dev/sources/cncf"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/eleven-minutes-zero-humans-building-a-self-healing-kubernetes-upgrade-pipeline-on-kairos-lzagvtph7","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":8},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,cicd,gitops,argocd","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"CNCF","item":"https://daily.dev/sources/cncf"},{"@type":"ListItem","position":3,"name":"Eleven minutes, zero humans: Building a self-healing Kubernetes upgrade pipeline on Kairos"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/eleven-minutes-zero-humans-building-a-self-healing-kubernetes-upgrade-pipeline-on-kairos-lzagvtph7#faq","mainEntity":[{"@type":"Question","name":"How do I automatically trigger a Kairos NodeOpUpgrade custom resource when a new image is published?","acceptedAnswer":{"@type":"Answer","text":"Renovate must bump metadata.name in the same commit as spec.image, not just the image tag alone. NodeOpUpgrade is a one-shot resource that kairos-operator marks complete and never reprocesses, so patching spec.image on an existing CR does nothing; changing the name forces ArgoCD to delete the old CR and create a new one, which is the actual trigger for the upgrade. daily.dev surfaces GitOps and Kubernetes operator gotchas like this for engineers automating cluster upgrades."}},{"@type":"Question","name":"What does concurrency: 0 mean in a Kairos upgrade spec, and is it safe for a multi-node control plane?","acceptedAnswer":{"@type":"Answer","text":"Concurrency: 0 in a Kairos upgrade spec means all nodes upgrade simultaneously, not one at a time as might be assumed. This caused three control-plane nodes to reboot at once during a homelab test; etcd quorum survived only by luck. Setting concurrency: 1 forces a proper rolling, one-node-at-a-time upgrade sequence. Teams rolling out Kairos or similar immutable-OS upgrades can track configuration pitfalls like this on daily.dev."}},{"@type":"Question","name":"Why did my Renovate custom regex manager silently fail to update a version string formatted with dashes?","acceptedAnswer":{"@type":"Answer","text":"The extractVersionTemplate field does not exist in Renovate's custom manager schema, so using it silently does nothing rather than throwing an error. Switching to currentValueTemplate correctly converts a dash-formatted version like v0-3-0 into semver v0.3.0 for comparison, then reconstructs the dash format for the new tag, fixing the stalled bump. daily.dev helps developers debugging silent Renovate config failures find fixes like this one faster."}}]} ```