--- title: AWS DevOps Agent Meets Chaos How I Wired an Autonomous... description: A hands-on walkthrough of AWS DevOps Agent, an autonomous SRE tool that investigates incidents without human intervention. The author provisions a full EKS lab... canonical: https://daily.dev/posts/aws-devops-agent-meets-chaos-how-i-wired-an-autonomous-sre-to-a-deliberately-broken-eks-cluster-xyqbexv4q twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: AWS DevOps Agent Meets Chaos How I Wired an Autonomous SRE to a Deliberately Broken EKS Cluster | daily.dev og:description: A hands-on walkthrough of AWS DevOps Agent, an autonomous SRE tool that investigates incidents without human intervention. The author provisions a full EKS lab... og:url: https://daily.dev/posts/aws-devops-agent-meets-chaos-how-i-wired-an-autonomous-sre-to-a-deliberately-broken-eks-cluster-xyqbexv4q og:image: https://api.daily.dev/og/posts/XYqBEXv4Q.png og:image:alt: AWS DevOps Agent Meets Chaos How I Wired an Autonomous SRE to a Deliberately Broken EKS Cluster og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # AWS DevOps Agent Meets Chaos How I Wired an Autonomous SRE to a Deliberately Broken EKS Cluster **[Medium](https://daily.dev/sources/medium_js)** · 11 min read · 1 upvotes · 0 comments ## Summary A hands-on walkthrough of AWS DevOps Agent, an autonomous SRE tool that investigates incidents without human intervention. The author provisions a full EKS lab with Terraform and runs four chaos scenarios: OOMKill loops, deployment regressions, Lambda timeouts, and a multi-service cascade failure. The agent correctly identifies root causes in all four cases by correlating CloudWatch telemetry, deployment history from GitHub, infrastructure configuration, and a topology map of dependencies. Standout capabilities include deployment correlation (matching alarm timestamps to Git commits in seconds), cascade failure triage via topology traversal, and proactive recommendations for missing PodDisruptionBudgets, AZ spread, and log retention gaps. The full Terraform module and chaos scenarios are available on GitHub. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does AWS DevOps Agent identify the root cause of a Kubernetes OOMKill loop? It correlates the CloudWatch memory alarm with EKS pod events showing OOMKilled as the termination reason, then pulls the deployment configuration to compare the pod's memory limit against the actual memory the process requests. In a tested scenario, a container requesting 256MB was capped at a 64Mi limit, and the agent recommended raising the limit to at least 512Mi while also flagging missing PodDisruptionBudgets and topology spread constraints. _daily.dev surfaces practical EKS incident-response writeups for engineers automating on-call debugging._ ### Can AWS DevOps Agent detect a bad Helm deployment as the cause of an incident? Yes, it correlates the timing of an ALB error rate alarm with recent deployment history from GitHub Actions, pulls the Helm values diff, and identifies the specific breaking change. In one test it found an invalid image tag causing ImagePullBackOff within two minutes, plus a secondary liveness probe path change that would have caused CrashLoopBackOff even with a working image. _Engineers evaluating deployment-correlation tooling can track results like this on daily.dev._ ### What IAM permissions does AWS DevOps Agent need to investigate an AWS environment? It requires a read-only execution role covering CloudWatch, EKS, EC2, Lambda, ECS, RDS, Resource Tagging, and X-Ray, with no write access, since the agent only produces mitigation plans for a human or coding agent to execute. The assume-role policy should include an aws:SourceAccount condition to prevent any AWS DevOps Agent in another account from assuming the role. _Teams scoping least-privilege IAM roles for AI agents can follow discussions like this on daily.dev._ ## Similar posts on daily.dev - [Automate root cause analysis across Datadog and Elasticsearch with AWS DevOps Agent](https://daily.dev/posts/automate-root-cause-analysis-across-datadog-and-elasticsearch-with-aws-devops-agent-pg6b3fzvz) · AWS · 0 upvotes · 0 comments - [AWS Announces General Availability of DevOps Agent for Automated Incident Investigation](https://daily.dev/posts/aws-announces-general-availability-of-devops-agent-for-automated-incident-investigation-o3foudjfb) · InfoQ · 0 upvotes · 0 comments - [Self-Healing Kubernetes Gets Real — and Risky: Running AI Agents on Amazon EKS](https://daily.dev/posts/self-healing-kubernetes-gets-real-and-risky-running-ai-agents-on-amazon-eks-0cqww64he) · Cloud Native Now · 0 upvotes · 0 comments - [How we built an autonomous SRE agent for Kubernetes](https://daily.dev/posts/how-we-built-an-autonomous-sre-agent-for-kubernetes-gxfkkntbg) · LangChain · 6 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#kubernetes](https://daily.dev/tags/kubernetes), [#terraform](https://daily.dev/tags/terraform) [View this post on daily.dev](https://daily.dev/posts/aws-devops-agent-meets-chaos-how-i-wired-an-autonomous-sre-to-a-deliberately-broken-eks-cluster-xyqbexv4q) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"AWS DevOps Agent Meets Chaos How I Wired an Autonomous SRE to a Deliberately Broken EKS Cluster","url":"https://daily.dev/posts/aws-devops-agent-meets-chaos-how-i-wired-an-autonomous-sre-to-a-deliberately-broken-eks-cluster-xyqbexv4q","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/aws-devops-agent-meets-chaos-how-i-wired-an-autonomous-sre-to-a-deliberately-broken-eks-cluster-xyqbexv4q"},"datePublished":"2026-07-20T07:33:13.417Z","dateModified":"2026-09-14T09:09:33.962Z","description":"A hands-on walkthrough of AWS DevOps Agent, an autonomous SRE tool that investigates incidents without human intervention. The author provisions a full EKS lab...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5f1923a3532e7f27ab072826ea0fc4ad?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5f1923a3532e7f27ab072826ea0fc4ad?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/aws-devops-agent-meets-chaos-how-i-wired-an-autonomous-sre-to-a-deliberately-broken-eks-cluster-xyqbexv4q","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,kubernetes,terraform","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"AWS DevOps Agent Meets Chaos How I Wired an Autonomous SRE to a Deliberately Broken EKS Cluster"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/aws-devops-agent-meets-chaos-how-i-wired-an-autonomous-sre-to-a-deliberately-broken-eks-cluster-xyqbexv4q#faq","mainEntity":[{"@type":"Question","name":"How does AWS DevOps Agent identify the root cause of a Kubernetes OOMKill loop?","acceptedAnswer":{"@type":"Answer","text":"It correlates the CloudWatch memory alarm with EKS pod events showing OOMKilled as the termination reason, then pulls the deployment configuration to compare the pod's memory limit against the actual memory the process requests. In a tested scenario, a container requesting 256MB was capped at a 64Mi limit, and the agent recommended raising the limit to at least 512Mi while also flagging missing PodDisruptionBudgets and topology spread constraints. daily.dev surfaces practical EKS incident-response writeups for engineers automating on-call debugging."}},{"@type":"Question","name":"Can AWS DevOps Agent detect a bad Helm deployment as the cause of an incident?","acceptedAnswer":{"@type":"Answer","text":"Yes, it correlates the timing of an ALB error rate alarm with recent deployment history from GitHub Actions, pulls the Helm values diff, and identifies the specific breaking change. In one test it found an invalid image tag causing ImagePullBackOff within two minutes, plus a secondary liveness probe path change that would have caused CrashLoopBackOff even with a working image. Engineers evaluating deployment-correlation tooling can track results like this on daily.dev."}},{"@type":"Question","name":"What IAM permissions does AWS DevOps Agent need to investigate an AWS environment?","acceptedAnswer":{"@type":"Answer","text":"It requires a read-only execution role covering CloudWatch, EKS, EC2, Lambda, ECS, RDS, Resource Tagging, and X-Ray, with no write access, since the agent only produces mitigation plans for a human or coding agent to execute. The assume-role policy should include an aws:SourceAccount condition to prevent any AWS DevOps Agent in another account from assuming the role. Teams scoping least-privilege IAM roles for AI agents can follow discussions like this on daily.dev."}}]} ```