A proof-of-concept incident response bot called Octopus Healer listens for Prometheus alerts on Kubernetes pods, sends pod logs and resource metrics to an AI model for a structured JSON diagnosis, then maps the model's chosen remediation type to a predefined Octopus Deploy runbook. The AI never executes commands directly; it only selects from an allowlisted set of remediation types (pod_restart, resource_increase, config_update, image_fix, deployment_rollback) and supplies parameters, while Octopus Deploy handles execution through existing permissions and approval workflows, with an operator approving in Slack before anything runs. The author covers design lessons: early attempts to have the model generate raw kubectl commands were unreliable, Config as Code (git-backed projects) added integration complexity, and the current in-memory approval store with a 30-minute TTL is not production-ready. Next steps include Slack webhook signature validation, a Helm chart, and adding deployment history context so the model can distinguish resource issues from regressions.
Table of contents
Why Octopus Deploy is the right execution layerThe full pipelineGiving an AI model the right contextTurning the model’s analysis into a runbookFully automated or operator-driven?What I learned — and what’s nextTags:Questions this post answers
How can I safely let an AI model suggest Kubernetes remediations without letting it run arbitrary kubectl commands?
Restrict the model to selecting from a predefined, allowlisted set of remediation types (like pod_restart, resource_increase, config_update, image_fix, deployment_rollback) and returning only structured JSON parameters, rather than generating executable shell commands. A separate execution layer, such as Octopus Deploy runbooks, then fills a reviewed template with those values and requires operator approval before running anything. daily.dev surfaces practical patterns like this for teams designing safer AI-assisted ops automation.
Why is letting an LLM generate raw kubectl commands for automated remediation unreliable?
Models can return kubectl flags that do not actually exist, or add explanatory text and Markdown formatting even when the prompt explicitly requests only a raw command. This unpredictability makes direct command generation unsafe for production automation, which is why parsing structured JSON output into a fixed set of validated remediation templates works better than trusting free-form generated commands. developers weighing how much control to hand an AI agent can compare approaches like this on daily.dev.
How do you link a Kubernetes deployment to its corresponding Octopus Deploy project for automated runbook execution?
Add an annotation directly on the Kubernetes Deployment manifest, such as octopus.com/project-id: Projects-42, since Prometheus alerts and Octopus Deploy have no inherent knowledge of each other. If the annotation is missing, a fallback can prompt an operator to manually pick the correct project, though restricting that picker to an allowlisted set is safer for production use. daily.dev helps engineers tracking Kubernetes and Octopus Deploy integration patterns stay informed.
10.8K Impressions1 Comment