--- title: Medium | daily.dev description: A detailed architectural walkthrough of a high-throughput streaming telemetry gateway built on Kubernetes, designed to handle millions of events per second... canonical: https://daily.dev/posts/medium-urqjx1dc2 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Medium | daily.dev og:description: A detailed architectural walkthrough of a high-throughput streaming telemetry gateway built on Kubernetes, designed to handle millions of events per second... og:url: https://daily.dev/posts/medium-urqjx1dc2 og:image: https://api.daily.dev/og/posts/urqjx1DC2.png og:image:alt: Medium og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Medium **[Medium](https://daily.dev/sources/medium_js)** · 9 min read · 0 upvotes · 0 comments ## Summary A detailed architectural walkthrough of a high-throughput streaming telemetry gateway built on Kubernetes, designed to handle millions of events per second from network devices. The system uses multi-tiered rate limiting that pushes backpressure to edge switches via HTTP 422 rejections with jittered exponential backoff, avoiding expensive cloud-side buffering. Numaflow MonoVertex is used to collapse multi-stage pipeline overhead into a single processing boundary communicating over Unix Domain Sockets. Dual-layer autoscaling covers both ingestion (HPA on CPU/memory) and processing (Numaflow's lag-based autoscaler) with asymmetric stabilization windows — fast scale-up (30s) and slow scale-down (5 min). In-flight authentication uses a two-level Redis cache plus gRPC fallback, and payloads are validated against a schema registry before enrichment with device context data. Key lessons include the necessity of jitter in backoff, the cost of symmetric HPA windows, and the importance of keeping enrichment caches warm. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does a telemetry gateway return HTTP 422 instead of just queuing excess traffic in the cloud? Returning HTTP 422 Unprocessable Entity when a pod hits its concurrency ceiling shifts the queue-buffering burden to the edge device's local hardware memory instead of expensive cloud memory. Network switches treat 422 as a transient, retriable condition, buffering events locally and retrying after a randomized exponential backoff, which smooths traffic spikes without dropping events or destabilizing the cluster. _daily.dev surfaces architecture patterns like edge backpressure for teams designing resilient ingestion pipelines._ ### What is Numaflow MonoVertex and when should I use it instead of a multi-stage streaming pipeline? MonoVertex is a Numaflow processing pattern that joins a User-Defined Source and User-Defined Sink into a single processing boundary communicating over Unix Domain Sockets, eliminating inter-step network hops. It fits single-pass read-enrich-write pipelines, cutting p99 latency and simplifying the failure surface, but a full Numaflow graph or a more capable framework like Flink is better once you need fan-out, branching, or multi-stage stateful operations. _Developers comparing streaming frameworks track tradeoffs like MonoVertex versus Flink on daily.dev._ ### Why should Kubernetes autoscaling use asymmetric stabilization windows for scale-up and scale-down? A narrow 30-second stabilization window allows replica counts to double within a minute during scale-up, absorbing spikes before they cause instability, while a longer 5-minute cooldown on scale-down prevents pods from thrashing during volatile, short-lived traffic surges. Symmetric windows caused pods to scale down too aggressively during brief lulls, and container startup latency became a visible component of p95 ingest times. _daily.dev helps engineers tuning HPA behavior compare real production configurations before shipping changes._ ## Similar posts on daily.dev - [Intuit’s Numaflow Abstracts Away Infrastructure for ML Engineers](https://daily.dev/posts/intuit-s-numaflow-abstracts-away-infrastructure-for-ml-engineers-gpehavg4h) · The New Stack · 0 upvotes · 0 comments - [Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned](https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7) · Netflix TechBlog · 26 upvotes · 0 comments - [How Netflix Scaled Its Real-Time Service Map](https://daily.dev/posts/how-netflix-scaled-its-real-time-service-map-u7xt8lptw) · InfoQ · 2 upvotes · 0 comments - [High Performance Rate Limiting at Databricks](https://daily.dev/posts/high-performance-rate-limiting-at-databricks-lln88cxlb) · ByteByteGo · 13 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#kafka](https://daily.dev/tags/kafka) [View this post on daily.dev](https://daily.dev/posts/medium-urqjx1dc2) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Medium","url":"https://daily.dev/posts/medium-urqjx1dc2","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/medium-urqjx1dc2"},"datePublished":"2026-07-21T19:33:02.931Z","dateModified":"2026-09-13T18:51:41.986Z","description":"A detailed architectural walkthrough of a high-throughput streaming telemetry gateway built on Kubernetes, designed to handle millions of events per second...","image":"https://media.daily.dev/image/upload/s--HRgLpUt6--/f_auto/v1722860399/public/Placeholder%2003","thumbnailUrl":"https://media.daily.dev/image/upload/s--HRgLpUt6--/f_auto/v1722860399/public/Placeholder%2003","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/medium-urqjx1dc2","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,kafka","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"Medium"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/medium-urqjx1dc2#faq","mainEntity":[{"@type":"Question","name":"Why does a telemetry gateway return HTTP 422 instead of just queuing excess traffic in the cloud?","acceptedAnswer":{"@type":"Answer","text":"Returning HTTP 422 Unprocessable Entity when a pod hits its concurrency ceiling shifts the queue-buffering burden to the edge device's local hardware memory instead of expensive cloud memory. Network switches treat 422 as a transient, retriable condition, buffering events locally and retrying after a randomized exponential backoff, which smooths traffic spikes without dropping events or destabilizing the cluster. daily.dev surfaces architecture patterns like edge backpressure for teams designing resilient ingestion pipelines."}},{"@type":"Question","name":"What is Numaflow MonoVertex and when should I use it instead of a multi-stage streaming pipeline?","acceptedAnswer":{"@type":"Answer","text":"MonoVertex is a Numaflow processing pattern that joins a User-Defined Source and User-Defined Sink into a single processing boundary communicating over Unix Domain Sockets, eliminating inter-step network hops. It fits single-pass read-enrich-write pipelines, cutting p99 latency and simplifying the failure surface, but a full Numaflow graph or a more capable framework like Flink is better once you need fan-out, branching, or multi-stage stateful operations. Developers comparing streaming frameworks track tradeoffs like MonoVertex versus Flink on daily.dev."}},{"@type":"Question","name":"Why should Kubernetes autoscaling use asymmetric stabilization windows for scale-up and scale-down?","acceptedAnswer":{"@type":"Answer","text":"A narrow 30-second stabilization window allows replica counts to double within a minute during scale-up, absorbing spikes before they cause instability, while a longer 5-minute cooldown on scale-down prevents pods from thrashing during volatile, short-lived traffic surges. Symmetric windows caused pods to scale down too aggressively during brief lulls, and container startup latency became a visible component of p95 ingest times. daily.dev helps engineers tuning HPA behavior compare real production configurations before shipping changes."}}]} ```