--- title: From 40 seconds to under 10: rebuilding incident... description: An engineering team at Atlassian describes rebuilding an automated incident-detection platform ('AutoHOT') from a Node.js aggregator into a stream-processing... canonical: https://daily.dev/posts/from-40-seconds-to-under-10-rebuilding-incident-detection-on-opentelemetry-apache-kafka-and-apach-owhrwpfy6 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: From 40 seconds to under 10: rebuilding incident detection on OpenTelemetry, Apache Kafka, and Apache Flink on Kubernetes | daily.dev og:description: An engineering team at Atlassian describes rebuilding an automated incident-detection platform ('AutoHOT') from a Node.js aggregator into a stream-processing... og:url: https://daily.dev/posts/from-40-seconds-to-under-10-rebuilding-incident-detection-on-opentelemetry-apache-kafka-and-apach-owhrwpfy6 og:image: https://api.daily.dev/og/posts/oWHrwPFY6.png og:image:alt: From 40 seconds to under 10: rebuilding incident detection on OpenTelemetry, Apache Kafka, and Apache Flink on Kubernetes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # From 40 seconds to under 10: rebuilding incident detection on OpenTelemetry, Apache Kafka, and Apache Flink on Kubernetes **[CNCF](https://daily.dev/sources/cncf)** · 18 min read · 9 upvotes · 0 comments ## Summary An engineering team at Atlassian describes rebuilding an automated incident-detection platform ('AutoHOT') from a Node.js aggregator into a stream-processing system built on Apache Kafka, a single Apache Flink job on Kubernetes, and OpenTelemetry. It covers architectural decisions (bus-side filtering, HyperLogLog for distinct-user counts, idempotent sinks, operator merge/split heuristics, autoscaler tuning), the decision engine that turns anomalies into paged incidents, and eighteen months of measured outcomes: event-to-metric latency dropped from 40+ seconds to under 10, monthly dashboard cost fell from $20K to $1K, and in-scope recall fluctuated between 60% and 86%. It candidly details failures such as silence-as-health blind spots, quantification lag during a major incident, read-time aggregation collapsing under concurrent load, and single-region detection blindness, ending with a list of lessons and next steps. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can HyperLogLog sketches be used to count distinct impacted users across tenants, shards, and regions in a streaming pipeline? Each 60-second tumbling window in the stream job carries an HLL sketch instead of raw user IDs, and a downstream API unions sketches at read time for whatever grouping is requested (tenant, shard, region, or global). This avoids double-counting from summing per-row counts across merges. The trade-off is format coupling between producer and consumer languages and roughly 1.5% counting error, which proved acceptable in practice. _Anyone designing cross-dimensional distinct-count metrics can find similar architecture write-ups curated on daily.dev._ ### Why does client-side operational telemetry fail to detect a hard-down database shard or service outage? Client-side telemetry only exists when a page successfully loads, so a hard-down dependency produces silence rather than error events, and a detection system built only to look for failures reads that silence as health. The fix requires volume-drop detectors, telemetry-freshness checks on the pipeline itself, and non-client signals like edge 5xx rates and synthetic probes to catch outages that never generate client-side events. _Engineers hardening observability against blind spots can track these detection design patterns on daily.dev._ ### What Apache Flink operator tuning reduced job restarts caused by autoscaling on Kubernetes? Setting the Flink Kubernetes operator's autoscaler to a 0.7 target utilization, a 0.3 boundary, vertex parallelism between 18 and 25, and a three-hour scale-down interval eliminated restarts entirely, down from around eight full job restarts a day. The team also added a deploy-pipeline guardrail that fails if a job restarts more than three times in ten minutes. _Teams tuning Flink-on-Kubernetes autoscaling can find related production configuration tips on daily.dev._ ## Similar posts on daily.dev - [Atlassian Rebuilds Metrics Pipeline Around OpenTelemetry at Massive Scale](https://daily.dev/posts/atlassian-rebuilds-metrics-pipeline-around-opentelemetry-at-massive-scale-iayb6ques) · InfoQ · 0 upvotes · 0 comments - [Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned](https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7) · Netflix TechBlog · 26 upvotes · 0 comments - [Real-Time AI Pipelines: Building a Kafka \+ Kubernetes \+ Databricks Stack for Agentic AI Systems](https://daily.dev/posts/real-time-ai-pipelines-building-a-kafka-kubernetes-databricks-stack-for-agentic-ai-systems-qoytpduhq) · Medium · 0 upvotes · 0 comments - [An End-to-End Cloud Native Observability Framework](https://daily.dev/posts/an-end-to-end-cloud-native-observability-framework-4gzq4zqow) · The New Stack · 0 upvotes · 0 comments - [Why Real-Time Stream Processing Beats Batch ETL for AI Data Freshness in 2026](https://daily.dev/posts/why-real-time-stream-processing-beats-batch-etl-for-ai-data-freshness-in-2026-c1hypylls) · Confluent Blog · 3 upvotes · 0 comments --- Tags: [#opentelemetry](https://daily.dev/tags/opentelemetry), [#apache-kafka](https://daily.dev/tags/apache-kafka), [#apache-flink](https://daily.dev/tags/apache-flink) [View this post on daily.dev](https://daily.dev/posts/from-40-seconds-to-under-10-rebuilding-incident-detection-on-opentelemetry-apache-kafka-and-apach-owhrwpfy6) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"From 40 seconds to under 10: rebuilding incident detection on OpenTelemetry, Apache Kafka, and Apache Flink on Kubernetes","url":"https://daily.dev/posts/from-40-seconds-to-under-10-rebuilding-incident-detection-on-opentelemetry-apache-kafka-and-apach-owhrwpfy6","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/from-40-seconds-to-under-10-rebuilding-incident-detection-on-opentelemetry-apache-kafka-and-apach-owhrwpfy6"},"datePublished":"2026-09-30T11:04:35.768Z","dateModified":"2026-09-30T11:05:03.877Z","description":"An engineering team at Atlassian describes rebuilding an automated incident-detection platform ('AutoHOT') from a Node.js aggregator into a stream-processing...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fa39ef5ddd3b3c19a0821c38862dc7ef?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fa39ef5ddd3b3c19a0821c38862dc7ef?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"CNCF","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"CNCF","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/55536a03390741eebd96d2f9b74d2d8d","url":"https://daily.dev/sources/cncf"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/from-40-seconds-to-under-10-rebuilding-incident-detection-on-opentelemetry-apache-kafka-and-apach-owhrwpfy6","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":9},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"opentelemetry,apache-kafka,apache-flink","timeRequired":"PT18M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"CNCF","item":"https://daily.dev/sources/cncf"},{"@type":"ListItem","position":3,"name":"From 40 seconds to under 10: rebuilding incident detection on OpenTelemetry, Apache Kafka, and Apache Flink on Kubernetes"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/from-40-seconds-to-under-10-rebuilding-incident-detection-on-opentelemetry-apache-kafka-and-apach-owhrwpfy6#faq","mainEntity":[{"@type":"Question","name":"How can HyperLogLog sketches be used to count distinct impacted users across tenants, shards, and regions in a streaming pipeline?","acceptedAnswer":{"@type":"Answer","text":"Each 60-second tumbling window in the stream job carries an HLL sketch instead of raw user IDs, and a downstream API unions sketches at read time for whatever grouping is requested (tenant, shard, region, or global). This avoids double-counting from summing per-row counts across merges. The trade-off is format coupling between producer and consumer languages and roughly 1.5% counting error, which proved acceptable in practice. Anyone designing cross-dimensional distinct-count metrics can find similar architecture write-ups curated on daily.dev."}},{"@type":"Question","name":"Why does client-side operational telemetry fail to detect a hard-down database shard or service outage?","acceptedAnswer":{"@type":"Answer","text":"Client-side telemetry only exists when a page successfully loads, so a hard-down dependency produces silence rather than error events, and a detection system built only to look for failures reads that silence as health. The fix requires volume-drop detectors, telemetry-freshness checks on the pipeline itself, and non-client signals like edge 5xx rates and synthetic probes to catch outages that never generate client-side events. Engineers hardening observability against blind spots can track these detection design patterns on daily.dev."}},{"@type":"Question","name":"What Apache Flink operator tuning reduced job restarts caused by autoscaling on Kubernetes?","acceptedAnswer":{"@type":"Answer","text":"Setting the Flink Kubernetes operator's autoscaler to a 0.7 target utilization, a 0.3 boundary, vertex parallelism between 18 and 25, and a three-hour scale-down interval eliminated restarts entirely, down from around eight full job restarts a day. The team also added a deploy-pipeline guardrail that fails if a job restarts more than three times in ten minutes. Teams tuning Flink-on-Kubernetes autoscaling can find related production configuration tips on daily.dev."}}]} ```