--- title: Why killing pods is not enough: Testing operator... description: Traditional chaos tools like Krkn or Litmus test pod restarts, which only validates Kubernetes' built-in Deployment controller — not your operator's... canonical: https://daily.dev/posts/why-killing-pods-is-not-enough-testing-operator-reconciliation-with-operator-chaos-wpoesrwhl twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why killing pods is not enough: Testing operator reconciliation with operator-chaos | daily.dev og:description: Traditional chaos tools like Krkn or Litmus test pod restarts, which only validates Kubernetes' built-in Deployment controller — not your operator's... og:url: https://daily.dev/posts/why-killing-pods-is-not-enough-testing-operator-reconciliation-with-operator-chaos-wpoesrwhl og:image: https://api.daily.dev/og/posts/WPOESrwhl.png og:image:alt: Why killing pods is not enough: Testing operator reconciliation with operator-chaos og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why killing pods is not enough: Testing operator reconciliation with operator-chaos **[Red Hat Developer](https://daily.dev/sources/rhdev)** · 11 min read · 1 upvotes · 0 comments ## Summary Traditional chaos tools like Krkn or Litmus test pod restarts, which only validates Kubernetes' built-in Deployment controller — not your operator's reconciliation logic. operator-chaos is an open source Go framework that fills this gap with 20 failure injection types across four categories (infrastructure, configuration, access control, lifecycle) and four injection modes: CLI black-box, HTTP transport-level, client.Client-level, and per-action pipeline interceptors. A knowledge model YAML file teaches the framework what resources your operator manages and what healthy state looks like. The framework runs a five-phase experiment lifecycle (pre-check, inject, observe, post-check, verdict) and produces structured verdicts: Resilient, Degraded, Failed, or Inconclusive. A live demo shows cert-manager passing 18/18 experiments. The post also covers a zero-dependency chaostransport module to avoid version conflicts, ConfigMap-driven runtime fault control, and a fuzz testing harness for edge-case discovery. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why don't tools like Krkn or Litmus catch Kubernetes operator reconciliation bugs? Those tools mainly test whether killed pods restart, which only verifies the built-in Kubernetes Deployment controller, not the operator's own reconciliation logic. Failures like a Deployment scaled to zero, a deleted Service, or a removed leader-election Lease often go undetected because nothing forces the operator to notice and repair that drift. _Anyone hardening operators against silent reconciliation gaps can find deeper chaos-testing approaches on daily.dev._ ### What are the four failure-injection modes in operator-chaos? CLI mode runs external ChaosExperiment YAML files against a live cluster with zero code changes; transport-level mode wraps http.RoundTripper via rest.Config.WrapTransport to intercept API server HTTP calls with three lines of code; client-level mode wraps client.Client to fail specific CRUD operations; and action-pipeline mode uses ActionInterceptor to fail or skip individual reconciler actions like deploy or garbage collection. _Engineers choosing a chaos-testing layer for their operator can compare these approaches on daily.dev._ ### How does the chaostransport module avoid dependency conflicts with different controller-runtime versions? The chaostransport sub-module has a go.mod with only two directives and zero dependencies on controller-runtime or any k8s.io library, requiring Go 1.22.0. This lets any operator, regardless of its pinned controller-runtime or k8s.io version, import it via go get without triggering a major dependency upgrade or transitive dependency conflicts. _Teams juggling pinned controller-runtime versions across operators can track dependency-safe tooling on daily.dev._ ## Similar posts on daily.dev - [What 429 chaos experiments taught us about Kubernetes operator resilience](https://daily.dev/posts/what-429-chaos-experiments-taught-us-about-kubernetes-operator-resilience-io2kr0h46) · Red Hat Developer · 3 upvotes · 0 comments - [Beyond pass or fail: The new era of chaos engineering](https://daily.dev/posts/beyond-pass-or-fail-the-new-era-of-chaos-engineering-wg3vdbblb) · Red Hat Developer · 0 upvotes · 0 comments - [Chaos Engineering Practices to Increase Confidence and Reliability](https://daily.dev/posts/chaos-engineering-practices-to-increase-confidence-and-reliability-k2l7mzk1y) · monday Engineering · 1 upvotes · 0 comments - [Kubernetes chaos engineering at scale: Krkn Operator Developer Preview in Red Hat Advanced Cluster Management](https://daily.dev/posts/kubernetes-chaos-engineering-at-scale-krkn-operator-developer-preview-in-red-hat-advanced-cluster-m-uaavrdrey) · Red Hat Developer · 0 upvotes · 0 comments - [Lessons from Chaos Tests: We Lost Customer Experience in a Service We Thought Was Low-Risk](https://daily.dev/posts/lessons-from-chaos-tests-we-lost-customer-experience-in-a-service-we-thought-was-low-risk-euayrw9lm) · Trendyol Tech · 0 upvotes · 0 comments --- Tags: [#testing](https://daily.dev/tags/testing), [#kubernetes](https://daily.dev/tags/kubernetes), [#golang](https://daily.dev/tags/golang) [View this post on daily.dev](https://daily.dev/posts/why-killing-pods-is-not-enough-testing-operator-reconciliation-with-operator-chaos-wpoesrwhl) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why killing pods is not enough: Testing operator reconciliation with operator-chaos","url":"https://daily.dev/posts/why-killing-pods-is-not-enough-testing-operator-reconciliation-with-operator-chaos-wpoesrwhl","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-killing-pods-is-not-enough-testing-operator-reconciliation-with-operator-chaos-wpoesrwhl"},"datePublished":"2026-07-11T03:40:53.248Z","dateModified":"2026-09-14T07:01:47.478Z","description":"Traditional chaos tools like Krkn or Litmus test pod restarts, which only validates Kubernetes' built-in Deployment controller — not your operator's...","image":"https://media.daily.dev/image/upload/s--ZrL_HSsR--/f_auto/v1722860399/public/Placeholder%2006","thumbnailUrl":"https://media.daily.dev/image/upload/s--ZrL_HSsR--/f_auto/v1722860399/public/Placeholder%2006","isAccessibleForFree":true,"articleSection":"Red Hat Developer","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Red Hat Developer","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/2a8895f3216c4900bdc455585a4ad3e4","url":"https://daily.dev/sources/rhdev"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/why-killing-pods-is-not-enough-testing-operator-reconciliation-with-operator-chaos-wpoesrwhl","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"testing,kubernetes,golang","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Red Hat Developer","item":"https://daily.dev/sources/rhdev"},{"@type":"ListItem","position":3,"name":"Why killing pods is not enough: Testing operator reconciliation with operator-chaos"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/why-killing-pods-is-not-enough-testing-operator-reconciliation-with-operator-chaos-wpoesrwhl#faq","mainEntity":[{"@type":"Question","name":"Why don't tools like Krkn or Litmus catch Kubernetes operator reconciliation bugs?","acceptedAnswer":{"@type":"Answer","text":"Those tools mainly test whether killed pods restart, which only verifies the built-in Kubernetes Deployment controller, not the operator's own reconciliation logic. Failures like a Deployment scaled to zero, a deleted Service, or a removed leader-election Lease often go undetected because nothing forces the operator to notice and repair that drift. Anyone hardening operators against silent reconciliation gaps can find deeper chaos-testing approaches on daily.dev."}},{"@type":"Question","name":"What are the four failure-injection modes in operator-chaos?","acceptedAnswer":{"@type":"Answer","text":"CLI mode runs external ChaosExperiment YAML files against a live cluster with zero code changes; transport-level mode wraps http.RoundTripper via rest.Config.WrapTransport to intercept API server HTTP calls with three lines of code; client-level mode wraps client.Client to fail specific CRUD operations; and action-pipeline mode uses ActionInterceptor to fail or skip individual reconciler actions like deploy or garbage collection. Engineers choosing a chaos-testing layer for their operator can compare these approaches on daily.dev."}},{"@type":"Question","name":"How does the chaostransport module avoid dependency conflicts with different controller-runtime versions?","acceptedAnswer":{"@type":"Answer","text":"The chaostransport sub-module has a go.mod with only two directives and zero dependencies on controller-runtime or any k8s.io library, requiring Go 1.22.0. This lets any operator, regardless of its pinned controller-runtime or k8s.io version, import it via go get without triggering a major dependency upgrade or transitive dependency conflicts. Teams juggling pinned controller-runtime versions across operators can track dependency-safe tooling on daily.dev."}}]} ```