--- title: Monitoring Kubernetes in Production | daily.dev description: Sharing real-world failures encountered running Kubernetes clusters on AWS and bare metal - including an etcd instability caused by high-latency network disks... canonical: https://daily.dev/posts/monitoring-kubernetes-in-production-bxqe1jltp twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Monitoring Kubernetes in Production | daily.dev og:description: Sharing real-world failures encountered running Kubernetes clusters on AWS and bare metal - including an etcd instability caused by high-latency network disks... og:url: https://daily.dev/posts/monitoring-kubernetes-in-production-bxqe1jltp og:image: https://api.daily.dev/og/posts/bxqE1jLtp.png og:image:alt: Monitoring Kubernetes in Production og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Monitoring Kubernetes in Production **[Teleport](https://daily.dev/sources/teleport)** · 7 min read · 0 upvotes · 0 comments ## Summary Sharing real-world failures encountered running Kubernetes clusters on AWS and bare metal - including an etcd instability caused by high-latency network disks that silently broke SkyDNS - the team explains why traditional monitoring tools like monit fall short for cluster-level health checks. They introduce Satellite, an open-sourced Serf-based gossip mesh of monitoring agents that run smoke tests against core Kubernetes components (etcd, scheduler, API server) and persist health history to SQLite or InfluxDB, remaining functional even during partial node/master outages. Current limitations include poor scaling beyond small clusters, with plans for supernode-style aggregators. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did my Kubernetes SkyDNS pod go into CrashLoopBackoff even though etcd looked healthy? High latency on network-attached disks can cause etcd to fail reads and writes even while it appears to be running and properly configured. This makes etcd inconsistently available as a Kubernetes Service, which breaks SkyDNS's connection to it after the first restart, even though pod and cluster status checks show everything as healthy. _daily.dev surfaces infrastructure debugging writeups like this for engineers chasing intermittent cluster failures._ ### How can I monitor overall Kubernetes cluster health beyond built-in application monitoring? A mesh of agent processes running on every cluster node, communicating over the Serf gossip protocol, can track the health of key components like etcd, the scheduler, and the API server by running lightweight smoke tests. Results sync peer-to-peer and persist to a backend such as SQLite or InfluxDB, so health data remains accessible even if the master or several nodes go down. _engineers weighing cluster monitoring approaches can track patterns like this via daily.dev._ ### What are the downsides of using monit for monitoring a Kubernetes cluster? Monit is a lean, battle-tested single-executable process daemon, but it is built for monitoring a single system, not a distributed cluster. It has a limited set of built-in tests, weak extensibility requiring custom scripts, and is very difficult to wire multiple instances together into a coherent, highly available network of collaborating monitoring agents. _daily.dev helps infrastructure teams comparing monitoring tools for distributed systems stay informed._ ## Similar posts on daily.dev - [Kubernetes Monitoring Tools: What Actually Works at Scale](https://daily.dev/posts/kubernetes-monitoring-tools-what-actually-works-at-scale-zdg0bbpod) · Last9 · 2 upvotes · 0 comments - [Production-Grade Kubernetes Monitoring with Prometheus, Grafana, and kube-state-metrics](https://daily.dev/posts/production-grade-kubernetes-monitoring-with-prometheus-grafana-and-kube-state-metrics-hdukjp3rk) · Medium · 2 upvotes · 0 comments - [Operating Kubernetes at scale: a few stories from running Amazon EKS](https://daily.dev/posts/operating-kubernetes-at-scale-a-few-stories-from-running-amazon-eks-vdcgnhxcm) · The New Stack · 1 upvotes · 0 comments - [Kubernetes Monitoring Tools: 14 Best Options by Category](https://daily.dev/posts/kubernetes-monitoring-tools-14-best-options-by-category-i5nyqv9sq) · finout · 0 upvotes · 0 comments - [k8s-1m Overview](https://daily.dev/posts/k8s-1m-overview-uj6kbjdzt) · Lobsters · 0 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/monitoring-kubernetes-in-production-bxqe1jltp) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Monitoring Kubernetes in Production","url":"https://daily.dev/posts/monitoring-kubernetes-in-production-bxqe1jltp","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/monitoring-kubernetes-in-production-bxqe1jltp"},"datePublished":"2026-08-23T12:23:30.141Z","dateModified":"2026-08-23T12:52:32.321Z","description":"Sharing real-world failures encountered running Kubernetes clusters on AWS and bare metal - including an etcd instability caused by high-latency network disks...","image":"https://media.daily.dev/image/upload/s--1KxV4ohY--/f_auto/v1722860400/public/Placeholder%2007","thumbnailUrl":"https://media.daily.dev/image/upload/s--1KxV4ohY--/f_auto/v1722860400/public/Placeholder%2007","isAccessibleForFree":true,"articleSection":"Teleport","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Teleport","logo":"https://media.daily.dev/image/upload/s--Dw-Bbw6O--/c_limit,w_256/f_auto,q_auto/v1787487494/logos/teleport?_a=BAMAMicg0","url":"https://daily.dev/sources/teleport"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/monitoring-kubernetes-in-production-bxqe1jltp","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,kubernetes,observability,distributed-systems","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Teleport","item":"https://daily.dev/sources/teleport"},{"@type":"ListItem","position":3,"name":"Monitoring Kubernetes in Production"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/monitoring-kubernetes-in-production-bxqe1jltp#faq","mainEntity":[{"@type":"Question","name":"Why did my Kubernetes SkyDNS pod go into CrashLoopBackoff even though etcd looked healthy?","acceptedAnswer":{"@type":"Answer","text":"High latency on network-attached disks can cause etcd to fail reads and writes even while it appears to be running and properly configured. This makes etcd inconsistently available as a Kubernetes Service, which breaks SkyDNS's connection to it after the first restart, even though pod and cluster status checks show everything as healthy. daily.dev surfaces infrastructure debugging writeups like this for engineers chasing intermittent cluster failures."}},{"@type":"Question","name":"How can I monitor overall Kubernetes cluster health beyond built-in application monitoring?","acceptedAnswer":{"@type":"Answer","text":"A mesh of agent processes running on every cluster node, communicating over the Serf gossip protocol, can track the health of key components like etcd, the scheduler, and the API server by running lightweight smoke tests. Results sync peer-to-peer and persist to a backend such as SQLite or InfluxDB, so health data remains accessible even if the master or several nodes go down. engineers weighing cluster monitoring approaches can track patterns like this via daily.dev."}},{"@type":"Question","name":"What are the downsides of using monit for monitoring a Kubernetes cluster?","acceptedAnswer":{"@type":"Answer","text":"Monit is a lean, battle-tested single-executable process daemon, but it is built for monitoring a single system, not a distributed cluster. It has a limited set of built-in tests, weak extensibility requiring custom scripts, and is very difficult to wire multiple instances together into a coherent, highly available network of collaborating monitoring agents. daily.dev helps infrastructure teams comparing monitoring tools for distributed systems stay informed."}}]} ```