--- title: When a validating webhook blocks the ConfigMap that... description: A Kubernetes incident postmortem where a ValidatingWebhookConfiguration with failurePolicy: Fail caused a deadlock: the webhook pod was crash-looping due to a... canonical: https://daily.dev/posts/when-a-validating-webhook-blocks-the-configmap-that-would-fix-it-kodx6zalj twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: When a validating webhook blocks the ConfigMap that would fix it | daily.dev og:description: A Kubernetes incident postmortem where a ValidatingWebhookConfiguration with failurePolicy: Fail caused a deadlock: the webhook pod was crash-looping due to a... og:url: https://daily.dev/posts/when-a-validating-webhook-blocks-the-configmap-that-would-fix-it-kodx6zalj og:image: https://api.daily.dev/og/posts/KOdX6ZalJ.png og:image:alt: When a validating webhook blocks the ConfigMap that would fix it og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # When a validating webhook blocks the ConfigMap that would fix it **[Medium](https://daily.dev/sources/medium_js)** · 11 min read · 0 upvotes · 0 comments ## Summary A Kubernetes incident postmortem where a ValidatingWebhookConfiguration with failurePolicy: Fail caused a deadlock: the webhook pod was crash-looping due to a misconfigured liveness probe path, blocking all ConfigMap writes in the namespace — including the credential fix needed to restore a downed profile service. The resolution involved temporarily patching failurePolicy to Ignore (preserving audit trail over a full delete), applying the credential ConfigMap, then discovering a stale env var override on the Deployment was shadowing the fix. After removing the override and fixing the probe path, the webhook was restored. Structural fixes included moving the webhook's own resources to a dedicated namespace with a namespaceSelector exclusion, and adding an objectSelector for break-glass labeled ConfigMaps to avoid future deadlocks. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why does kubectl patch on a ConfigMap fail with a webhook connection refused error instead of a normal validation error This happens when a ValidatingWebhookConfiguration with failurePolicy set to Fail points at a webhook pod that is down or crash-looping, so every write in the namespace fails closed. The error surfaces as a webhook call failure (connection refused or timeout) rather than a validation rejection, because the admission controller cannot reach the webhook service at all. _Teams debugging webhook deadlocks like this compare recovery approaches on daily.dev before they hit production._ ### should I delete a ValidatingWebhookConfiguration or patch failurePolicy to Ignore to unblock writes during an incident Patching failurePolicy from Fail to Ignore is the smaller blast radius option: the webhook object keeps its identity and resourceVersion lineage, and the audit log shows one field change and one field change back, rather than a delete and recreate. Deleting is a cleaner cut but creates a louder audit event and a window where out-of-policy resources could slip through unnoticed. _Engineers weighing admission-control tradeoffs during incidents can track this kind of guidance on daily.dev._ ### why is my Kubernetes pod still using old database credentials after I updated the ConfigMap A Deployment-level environment variable can shadow a ConfigMap-sourced value if application code checks a raw env override before falling back to the ConfigMap key. In one documented case, a break-glass override variable set during a migration drill six weeks earlier silently took precedence over MONGODB_URI sourced from the ConfigMap, making a correct credential rotation appear to have no effect. _Anyone chasing stale-config mysteries after a credential rotation can find similar debugging writeups on daily.dev._ ## Similar posts on daily.dev - [How Kubernetes Health Checks Brought Down a Payment Service](https://daily.dev/posts/how-kubernetes-health-checks-brought-down-a-payment-service-jxkwnhtv6) · Medium · 12 upvotes · 0 comments - [CrashLoopBackOff in Kubernetes: The Real Causes and How We Fix It](https://daily.dev/posts/crashloopbackoff-in-kubernetes-the-real-causes-and-how-we-fix-it-odapgy48g) · Cast AI · 3 upvotes · 1 comments - [How to Debug a Stuck Kubernetes Rollout with a Hands-On Lab](https://daily.dev/posts/how-to-debug-a-stuck-kubernetes-rollout-with-a-hands-on-lab-og5k6586y) · freeCodeCamp · 8 upvotes · 1 comments - [A Missing Binary Turned a Kubernetes Liveness Probe Into a Restart Loop](https://daily.dev/posts/a-missing-binary-turned-a-kubernetes-liveness-probe-into-a-restart-loop-d2orncubs) · Cloud Native Now · 27 upvotes · 2 comments - [Protect your Kubernetes Operator from OOMKill](https://daily.dev/posts/protect-your-kubernetes-operator-from-oomkill-to0zvoqps) · Red Hat Developer · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#gitops](https://daily.dev/tags/gitops) [View this post on daily.dev](https://daily.dev/posts/when-a-validating-webhook-blocks-the-configmap-that-would-fix-it-kodx6zalj) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"When a validating webhook blocks the ConfigMap that would fix it","url":"https://daily.dev/posts/when-a-validating-webhook-blocks-the-configmap-that-would-fix-it-kodx6zalj","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/when-a-validating-webhook-blocks-the-configmap-that-would-fix-it-kodx6zalj"},"datePublished":"2026-07-16T19:32:57.610Z","dateModified":"2026-09-14T07:01:48.368Z","description":"A Kubernetes incident postmortem where a ValidatingWebhookConfiguration with failurePolicy: Fail caused a deadlock: the webhook pod was crash-looping due to a...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4a01c46650688e9f82ee8a01446b3588?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4a01c46650688e9f82ee8a01446b3588?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/when-a-validating-webhook-blocks-the-configmap-that-would-fix-it-kodx6zalj","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,gitops","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"When a validating webhook blocks the ConfigMap that would fix it"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/when-a-validating-webhook-blocks-the-configmap-that-would-fix-it-kodx6zalj#faq","mainEntity":[{"@type":"Question","name":"why does kubectl patch on a ConfigMap fail with a webhook connection refused error instead of a normal validation error","acceptedAnswer":{"@type":"Answer","text":"This happens when a ValidatingWebhookConfiguration with failurePolicy set to Fail points at a webhook pod that is down or crash-looping, so every write in the namespace fails closed. The error surfaces as a webhook call failure (connection refused or timeout) rather than a validation rejection, because the admission controller cannot reach the webhook service at all. Teams debugging webhook deadlocks like this compare recovery approaches on daily.dev before they hit production."}},{"@type":"Question","name":"should I delete a ValidatingWebhookConfiguration or patch failurePolicy to Ignore to unblock writes during an incident","acceptedAnswer":{"@type":"Answer","text":"Patching failurePolicy from Fail to Ignore is the smaller blast radius option: the webhook object keeps its identity and resourceVersion lineage, and the audit log shows one field change and one field change back, rather than a delete and recreate. Deleting is a cleaner cut but creates a louder audit event and a window where out-of-policy resources could slip through unnoticed. Engineers weighing admission-control tradeoffs during incidents can track this kind of guidance on daily.dev."}},{"@type":"Question","name":"why is my Kubernetes pod still using old database credentials after I updated the ConfigMap","acceptedAnswer":{"@type":"Answer","text":"A Deployment-level environment variable can shadow a ConfigMap-sourced value if application code checks a raw env override before falling back to the ConfigMap key. In one documented case, a break-glass override variable set during a migration drill six weeks earlier silently took precedence over MONGODB_URI sourced from the ConfigMap, making a correct credential rotation appear to have no effect. Anyone chasing stale-config mysteries after a credential rotation can find similar debugging writeups on daily.dev."}}]} ```