--- title: The Horrors of Upgrading Etcd Beneath Kubernetes | daily.dev description: An interview with a platform engineer discusses the challenges of performing autonomous, in-place etcd upgrades beneath air-gapped Kubernetes clusters. Topics... canonical: https://daily.dev/posts/the-horrors-of-upgrading-etcd-beneath-kubernetes-rqyjrdnco twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The Horrors of Upgrading Etcd Beneath Kubernetes | daily.dev og:description: An interview with a platform engineer discusses the challenges of performing autonomous, in-place etcd upgrades beneath air-gapped Kubernetes clusters. Topics... og:url: https://daily.dev/posts/the-horrors-of-upgrading-etcd-beneath-kubernetes-rqyjrdnco og:image: https://api.daily.dev/og/posts/RQYjrDNcO.png og:image:alt: The Horrors of Upgrading Etcd Beneath Kubernetes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The Horrors of Upgrading Etcd Beneath Kubernetes **[Teleport](https://daily.dev/sources/teleport)** · 23 min read · 0 upvotes · 0 comments ## Summary An interview with a platform engineer discusses the challenges of performing autonomous, in-place etcd upgrades beneath air-gapped Kubernetes clusters. Topics covered include why stepped upgrade paths make rollbacks hard, the catch-22 of using etcd itself as the coordination store during its own upgrade, why full Kubernetes API backup/restore (e.g. via Ark) was rejected in favor of a lower-level etcd data copy approach, using an external boltdb-based coordinator to survive etcd downtime, and quirks like Raft index changes confusing API watchers after restore. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why is it hard to roll back an etcd upgrade from v2 to v3? Etcd v3 uses a different API and storage backend than v2, so upgraded data isn't accessible through the v3 API without migration, and etcd's own upstream documentation recommends performing that migration as an offline process. There is no clean rolling rollback path; a failed in-place upgrade typically requires shutting down the entire cluster and restoring all data directories from backup. _Teams planning an etcd v2 to v3 migration can track real-world upgrade pitfalls like this on daily.dev._ ### How can Kubernetes clusters be upgraded without exposing etcd downtime to the cluster's coordination process? One approach is to avoid depending on etcd itself as the coordination mechanism during the upgrade, since etcd gets taken offline as part of the process. Instead, an external coordinator process can track upgrade steps in a local boltdb key-value store on each node, so the automation keeps running even while etcd is unavailable, then resumes normal state tracking once etcd comes back online. _Engineers automating cluster upgrades can follow practical coordination patterns like this via daily.dev._ ### Why would Kubernetes API watchers see unexpected behavior after an etcd backup and restore? Etcd assigns a Raft index to objects that acts like an internal pointer, and after a backup-and-restore cycle that index changes, so any listener not expecting etcd to disappear can get confused when it reappears with a different ID. Kubernetes documentation only says not to assume specific ID behavior, so third-party watchers relying on ever-increasing IDs may behave oddly. _Developers debugging odd Kubernetes watch behavior after cluster maintenance can find similar edge cases on daily.dev._ ## Similar posts on daily.dev - [Operating Kubernetes at scale: a few stories from running Amazon EKS](https://daily.dev/posts/operating-kubernetes-at-scale-a-few-stories-from-running-amazon-eks-vdcgnhxcm) · The New Stack · 1 upvotes · 0 comments - [k8s-1m Overview](https://daily.dev/posts/k8s-1m-overview-uj6kbjdzt) · Lobsters · 0 upvotes · 0 comments - [Set up a CI framework using Red Hat Ansible Automation Platform, Podman, and Horreum](https://daily.dev/posts/set-up-a-ci-framework-using-red-hat-ansible-automation-platform-podman-and-horreum-cre8xynwd) · Red Hat Developer · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/the-horrors-of-upgrading-etcd-beneath-kubernetes-rqyjrdnco) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The Horrors of Upgrading Etcd Beneath Kubernetes","url":"https://daily.dev/posts/the-horrors-of-upgrading-etcd-beneath-kubernetes-rqyjrdnco","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-horrors-of-upgrading-etcd-beneath-kubernetes-rqyjrdnco"},"datePublished":"2026-08-23T12:23:28.920Z","dateModified":"2026-08-23T12:57:08.635Z","description":"An interview with a platform engineer discusses the challenges of performing autonomous, in-place etcd upgrades beneath air-gapped Kubernetes clusters. Topics...","image":"https://media.daily.dev/image/upload/s--CxzD6vbw--/f_auto/v1722860399/public/Placeholder%2005","thumbnailUrl":"https://media.daily.dev/image/upload/s--CxzD6vbw--/f_auto/v1722860399/public/Placeholder%2005","isAccessibleForFree":true,"articleSection":"Teleport","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Teleport","logo":"https://media.daily.dev/image/upload/s--Dw-Bbw6O--/c_limit,w_256/f_auto,q_auto/v1787487494/logos/teleport?_a=BAMAMicg0","url":"https://daily.dev/sources/teleport"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-horrors-of-upgrading-etcd-beneath-kubernetes-rqyjrdnco","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,distributed-systems","timeRequired":"PT23M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Teleport","item":"https://daily.dev/sources/teleport"},{"@type":"ListItem","position":3,"name":"The Horrors of Upgrading Etcd Beneath Kubernetes"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-horrors-of-upgrading-etcd-beneath-kubernetes-rqyjrdnco#faq","mainEntity":[{"@type":"Question","name":"Why is it hard to roll back an etcd upgrade from v2 to v3?","acceptedAnswer":{"@type":"Answer","text":"Etcd v3 uses a different API and storage backend than v2, so upgraded data isn't accessible through the v3 API without migration, and etcd's own upstream documentation recommends performing that migration as an offline process. There is no clean rolling rollback path; a failed in-place upgrade typically requires shutting down the entire cluster and restoring all data directories from backup. Teams planning an etcd v2 to v3 migration can track real-world upgrade pitfalls like this on daily.dev."}},{"@type":"Question","name":"How can Kubernetes clusters be upgraded without exposing etcd downtime to the cluster's coordination process?","acceptedAnswer":{"@type":"Answer","text":"One approach is to avoid depending on etcd itself as the coordination mechanism during the upgrade, since etcd gets taken offline as part of the process. Instead, an external coordinator process can track upgrade steps in a local boltdb key-value store on each node, so the automation keeps running even while etcd is unavailable, then resumes normal state tracking once etcd comes back online. Engineers automating cluster upgrades can follow practical coordination patterns like this via daily.dev."}},{"@type":"Question","name":"Why would Kubernetes API watchers see unexpected behavior after an etcd backup and restore?","acceptedAnswer":{"@type":"Answer","text":"Etcd assigns a Raft index to objects that acts like an internal pointer, and after a backup-and-restore cycle that index changes, so any listener not expecting etcd to disappear can get confused when it reappears with a different ID. Kubernetes documentation only says not to assume specific ID behavior, so third-party watchers relying on ever-increasing IDs may behave oddly. Developers debugging odd Kubernetes watch behavior after cluster maintenance can find similar edge cases on daily.dev."}}]} ```