monday.com's engineering team built Observability Guard, a centralized observability platform integrated into their Internal Developer Portal (Sphera), to standardize alerting and log management across 600+ services and 10 different monitoring vendors. The system provides 107 alert templates covering Kubernetes infrastructure, automated resource discovery for per-resource thresholds, and MAD-based (Median Absolute Deviation) threshold calculation using 14-day baselines with k=3 for warnings and k=5 for critical alerts. Log management is handled through tiered storage with per-subsystem controls that propagate instantly. Beyond helping human engineers, the standardized alert structure creates a clean data contract for AI SRE agents, enabling accurate incident correlation. After over a year as opt-in, mandatory alerts were introduced for high-confidence signals.
Table of contents
Scaling the Nightmare to 600+ servicesWhy Good Observability is HardFood for AI SRE AgentsMeet the Observability GuardAlert TemplatesResource DiscoveryStop the GuessworkLog ManagementAdoptionGroundwork for Mandatory AlertsWhat It Gave UsThe New Normal at monday.comTurning Tribal into InstitutionalQuestions this post answers
How do you automatically calculate alerting thresholds without manual guesswork?
Median Absolute Deviation (MAD) is a robust method for auto-calculating alert thresholds. It calculates the median metric value over a 14-day baseline, measures the median of deviations from that baseline, then sets thresholds at k*MAD — k=3 for warnings and k=5 for critical alerts. MAD is preferred over standard deviation because a single spike cannot skew the baseline, giving a truer picture of normal behavior. Engineers tuning alert thresholds across many services track approaches like this on daily.dev.
How can I handle per-resource alerting thresholds for SQS queues without a one-size-fits-all rule?
A blanket wildcard alert covers all queues under a single baseline rule, while per-queue overrides let engineers set custom thresholds for specific queues. When an override is applied, that queue is automatically excluded from the blanket alert to prevent double-alerting. This pattern catches both large-queue breaches via the blanket rule and small-queue anomalies via targeted overrides. Teams managing queue-heavy architectures share threshold strategies like this on daily.dev.
What alert templates should every Kubernetes service have as a baseline?
A solid Kubernetes alerting baseline covers pod and ArgoCD app status, CPU and memory resource usage, HPA approaching limits, and CronJob issues. The kube-prometheus-stack defaults serve as a useful starting point. Beyond infrastructure signals, services depending on queues, databases, or caches need a second layer of dependency-specific alerts before any custom business-logic monitors are added. Platform engineers building Kubernetes alert standards find real-world implementations like this on daily.dev.
14.6K Impressions1 Comment