--- title: Using Prometheus and Grafana to Debug Kubernetes... description: A practical guide to debugging Kubernetes production incidents using Prometheus and Grafana. Covers an 8-step investigation workflow: checking cluster health,... canonical: https://daily.dev/posts/using-prometheus-and-grafana-to-debug-kubernetes-production-incidents-e7nzkq1vj twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Using Prometheus and Grafana to Debug Kubernetes Production Incidents | daily.dev og:description: A practical guide to debugging Kubernetes production incidents using Prometheus and Grafana. Covers an 8-step investigation workflow: checking cluster health,... og:url: https://daily.dev/posts/using-prometheus-and-grafana-to-debug-kubernetes-production-incidents-e7nzkq1vj og:image: https://api.daily.dev/og/posts/e7NZKq1VJ.png og:image:alt: Using Prometheus and Grafana to Debug Kubernetes Production Incidents og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Using Prometheus and Grafana to Debug Kubernetes Production Incidents **[Medium](https://daily.dev/sources/medium_js)** · 8 min read · 1 upvotes · 0 comments ## Summary A practical guide to debugging Kubernetes production incidents using Prometheus and Grafana. Covers an 8-step investigation workflow: checking cluster health, correlating recent changes, reviewing the Four Golden Signals (latency, traffic, errors, saturation), inspecting pod and deployment status, analyzing node resource pressure, identifying scheduling issues, monitoring external dependencies, and using time correlation across metrics. Includes real-world examples showing how metrics reveal root causes that logs alone miss, common incident types with specific metrics to check, dashboard organization strategies, and best practices like adding deployment annotations and using consistent labels. The core insight is that metrics should guide investigation before diving into logs. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What order should I check metrics in when debugging a Kubernetes production incident? Start with cluster health (node readiness, API server, scheduler), then the four golden signals (latency, traffic, errors, saturation), then pod and deployment status using kube-state-metrics, then node resource pressure, then external dependencies like databases, and only then check application logs. Metrics narrow down where to look; logs explain what happened once you've located the component. _Debugging Kubernetes outages under pressure is easier with this kind of structured workflow saved on daily.dev._ ### How can I tell if a Kubernetes production incident was caused by a recent deployment versus a traffic spike? Overlay deployment annotations on Grafana dashboards alongside metrics like CPU, error rate, and latency on the same timeline. If a spike in errors or CPU starts exactly when a Deployment rolled out rather than when traffic increased, the release itself is the likely cause, not increased load. Time correlation across metrics exposes this cause-and-effect that isolated graphs miss. _Teams tracing incidents back to a bad rollout can find workflows like this via daily.dev._ ### Why would a Kubernetes deployment show fewer available replicas than desired even though CPU and memory look normal? Pods may be failing to become healthy due to failed readiness probes, image pull failures, scheduling problems, or resource constraints, even when overall CPU and memory metrics appear fine. Comparing desired replicas (for example 8) against available replicas (for example 5) points directly to which pods never started successfully, prompting investigation into probes, image pulls, or scheduling. _daily.dev surfaces practical debugging patterns for engineers chasing down stuck Kubernetes deployments._ ## Similar posts on daily.dev - [Production-Grade Kubernetes Monitoring with Prometheus, Grafana, and kube-state-metrics](https://daily.dev/posts/production-grade-kubernetes-monitoring-with-prometheus-grafana-and-kube-state-metrics-hdukjp3rk) · Medium · 2 upvotes · 0 comments - [Building a Zero-to-Hero Prometheus Dashboard for Kubernetes](https://daily.dev/posts/building-a-zero-to-hero-prometheus-dashboard-for-kubernetes-qhetrpyoj) · Medium · 0 upvotes · 0 comments - [Kubernetes Monitoring Tools: What Actually Works at Scale](https://daily.dev/posts/kubernetes-monitoring-tools-what-actually-works-at-scale-zdg0bbpod) · Last9 · 2 upvotes · 0 comments - [Kubernetes Metrics: Types, Tools, & Monitoring Guide](https://daily.dev/posts/kubernetes-metrics-types-tools-monitoring-guide-vfxngka7f) · Spacelift · 4 upvotes · 0 comments - [Kubernetes Monitoring Tools: 14 Best Options by Category](https://daily.dev/posts/kubernetes-monitoring-tools-14-best-options-by-category-i5nyqv9sq) · finout · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#grafana](https://daily.dev/tags/grafana), [#prometheus](https://daily.dev/tags/prometheus) [View this post on daily.dev](https://daily.dev/posts/using-prometheus-and-grafana-to-debug-kubernetes-production-incidents-e7nzkq1vj) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Using Prometheus and Grafana to Debug Kubernetes Production Incidents","url":"https://daily.dev/posts/using-prometheus-and-grafana-to-debug-kubernetes-production-incidents-e7nzkq1vj","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/using-prometheus-and-grafana-to-debug-kubernetes-production-incidents-e7nzkq1vj"},"datePublished":"2026-07-21T19:34:02.576Z","dateModified":"2026-09-13T20:29:46.108Z","description":"A practical guide to debugging Kubernetes production incidents using Prometheus and Grafana. Covers an 8-step investigation workflow: checking cluster health,...","image":"https://media.daily.dev/image/upload/s--HRgLpUt6--/f_auto/v1722860399/public/Placeholder%2003","thumbnailUrl":"https://media.daily.dev/image/upload/s--HRgLpUt6--/f_auto/v1722860399/public/Placeholder%2003","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/using-prometheus-and-grafana-to-debug-kubernetes-production-incidents-e7nzkq1vj","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,observability,grafana,prometheus","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"Using Prometheus and Grafana to Debug Kubernetes Production Incidents"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/using-prometheus-and-grafana-to-debug-kubernetes-production-incidents-e7nzkq1vj#faq","mainEntity":[{"@type":"Question","name":"What order should I check metrics in when debugging a Kubernetes production incident?","acceptedAnswer":{"@type":"Answer","text":"Start with cluster health (node readiness, API server, scheduler), then the four golden signals (latency, traffic, errors, saturation), then pod and deployment status using kube-state-metrics, then node resource pressure, then external dependencies like databases, and only then check application logs. Metrics narrow down where to look; logs explain what happened once you've located the component. Debugging Kubernetes outages under pressure is easier with this kind of structured workflow saved on daily.dev."}},{"@type":"Question","name":"How can I tell if a Kubernetes production incident was caused by a recent deployment versus a traffic spike?","acceptedAnswer":{"@type":"Answer","text":"Overlay deployment annotations on Grafana dashboards alongside metrics like CPU, error rate, and latency on the same timeline. If a spike in errors or CPU starts exactly when a Deployment rolled out rather than when traffic increased, the release itself is the likely cause, not increased load. Time correlation across metrics exposes this cause-and-effect that isolated graphs miss. Teams tracing incidents back to a bad rollout can find workflows like this via daily.dev."}},{"@type":"Question","name":"Why would a Kubernetes deployment show fewer available replicas than desired even though CPU and memory look normal?","acceptedAnswer":{"@type":"Answer","text":"Pods may be failing to become healthy due to failed readiness probes, image pull failures, scheduling problems, or resource constraints, even when overall CPU and memory metrics appear fine. Comparing desired replicas (for example 8) against available replicas (for example 5) points directly to which pods never started successfully, prompting investigation into probes, image pulls, or scheduling. daily.dev surfaces practical debugging patterns for engineers chasing down stuck Kubernetes deployments."}}]} ```