MetalBear
Read post

Auto-Verifying Your AI-SRE’s Fixes (Part II): HolmesGPT End-to-End on a Real Cluster

A hands-on walkthrough of an AI-SRE verification loop running end-to-end on a real Kubernetes cluster using HolmesGPT and mirrord. Two bugs are planted in a Python checkout service: an error-rate bug (item-3 ValueError causing 500s) and a latency bug (missing client-side timeout causing p99 > 300ms). HolmesGPT investigates Alertmanager alerts, a Claude wrapper converts the report into a code patch, and a verifier runs the patched code via mirrord exec against the live cluster. Scenario 1 earns a PASS verdict (error rate drops from 10% to 0%). Scenario 2 earns a REJECT verdict because caching improves p50 by 99.8% but p99 remains at 2010ms, still violating the 300ms SLO. The key insight is that the verifier catches fixes that look good superficially but don't actually clear the alert condition. Full code (~500 lines Python) is available on GitHub.

    #kubernetes
Jun 23•7m read time•From metalbear.com
Post cover image
Table of contents
The demo cluster #Scenario 1: error-rate alert (PASS) #Scenario 2: latency alert (REJECT) #What the two scenarios show #Run it yourself #Frequently asked questions #
7.3K Impressions
MetalBear's image
MetalBear

33 Followers

•

611 Upvotes

Would you recommend this post?

Copy link
WhatsApp
Facebook
X
New Squad
  • © 2026 Daily Dev Ltd.
  • Guidelines
  • Explore
  • Tags
  • Sources
  • Squads
  • Leaderboard