--- title: "Fueling the AI SRE: How monday.com Standardized Observability for 600+ Services" url: https://daily.dev/posts/fueling-the-ai-sre-how-monday-com-standardized-observability-for-600-services-1kvwmav54 source_url: https://engineering.monday.com/fueling-the-ai-sre-how-monday-com-standardized-observability-for-600-services type: article source: "monday Engineering" published: 2026-08-10T10:07:17.033Z updated: 2026-08-10T10:07:50.464Z tags: ["kubernetes", "observability", "opentelemetry"] reading_time: 13 upvotes: 7 comments: 1 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Fueling the AI SRE: How monday.com Standardized Observability for 600+ Services **[monday Engineering](https://daily.dev/sources/monday)** · 13 min read · 7 upvotes · 1 comments ## Summary monday.com's engineering team built Observability Guard, a centralized observability platform integrated into their Internal Developer Portal (Sphera), to standardize alerting and log management across 600+ services and 10 different monitoring vendors. The system provides 107 alert templates covering Kubernetes infrastructure, automated resource discovery for per-resource thresholds, and MAD-based (Median Absolute Deviation) threshold calculation using 14-day baselines with k=3 for warnings and k=5 for critical alerts. Log management is handled through tiered storage with per-subsystem controls that propagate instantly. Beyond helping human engineers, the standardized alert structure creates a clean data contract for AI SRE agents, enabling accurate incident correlation. After over a year as opt-in, mandatory alerts were introduced for high-confidence signals. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do you automatically calculate alerting thresholds without manual guesswork? Median Absolute Deviation (MAD) is a robust method for auto-calculating alert thresholds. It calculates the median metric value over a 14-day baseline, measures the median of deviations from that baseline, then sets thresholds at k*MAD — k=3 for warnings and k=5 for critical alerts. MAD is preferred over standard deviation because a single spike cannot skew the baseline, giving a truer picture of normal behavior. _Engineers tuning alert thresholds across many services track approaches like this on daily.dev._ ### How can I handle per-resource alerting thresholds for SQS queues without a one-size-fits-all rule? A blanket wildcard alert covers all queues under a single baseline rule, while per-queue overrides let engineers set custom thresholds for specific queues. When an override is applied, that queue is automatically excluded from the blanket alert to prevent double-alerting. This pattern catches both large-queue breaches via the blanket rule and small-queue anomalies via targeted overrides. _Teams managing queue-heavy architectures share threshold strategies like this on daily.dev._ ### What alert templates should every Kubernetes service have as a baseline? A solid Kubernetes alerting baseline covers pod and ArgoCD app status, CPU and memory resource usage, HPA approaching limits, and CronJob issues. The kube-prometheus-stack defaults serve as a useful starting point. Beyond infrastructure signals, services depending on queues, databases, or caches need a second layer of dependency-specific alerts before any custom business-logic monitors are added. _Platform engineers building Kubernetes alert standards find real-world implementations like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@trevorsuna** · 0 upvotes > Standardizing alert shape before adding an AI layer feels like the real win. Consistent ownership, severity, runbooks, and resource context make automation safer and help humans too. The MAD thresholds are interesting, though I’d still want explicit handling for seasonal workloads and brand-new services. ## Similar posts on daily.dev - [Observability overload is drowning engineers](https://daily.dev/posts/observability-overload-is-drowning-engineers-fes9rhxs9) · The New Stack · 1 upvotes · 0 comments - [Self-service observability: Empower engineers to get the most value out of your observability data](https://daily.dev/posts/self-service-observability-empower-engineers-to-get-the-most-value-out-of-your-observability-data-tyvkgwa58) · Dynatrace · 1 upvotes · 0 comments - [Co-Developing an AI Native Observability Platform](https://daily.dev/posts/co-developing-an-ai-native-observability-platform-3ofczs5um) · DevOps.com · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#opentelemetry](https://daily.dev/tags/opentelemetry) [View this post on daily.dev](https://daily.dev/posts/fueling-the-ai-sre-how-monday-com-standardized-observability-for-600-services-1kvwmav54)