--- title: How to scale Alloy as a central telemetry gateway:... description: Grafana Labs' Professional Services team shares how they sized, load tested, and now run a production Alloy deployment as a centralized telemetry gateway for... canonical: https://daily.dev/posts/how-to-scale-alloy-as-a-central-telemetry-gateway-capacity-planning-load-testing-and-production-l-a0uuqqpox twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How to scale Alloy as a central telemetry gateway: capacity planning, load testing, and production lessons | daily.dev og:description: Grafana Labs' Professional Services team shares how they sized, load tested, and now run a production Alloy deployment as a centralized telemetry gateway for... og:url: https://daily.dev/posts/how-to-scale-alloy-as-a-central-telemetry-gateway-capacity-planning-load-testing-and-production-l-a0uuqqpox og:image: https://api.daily.dev/og/posts/a0UUqQpoX.png og:image:alt: How to scale Alloy as a central telemetry gateway: capacity planning, load testing, and production lessons og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How to scale Alloy as a central telemetry gateway: capacity planning, load testing, and production lessons **[Grafana Labs](https://daily.dev/sources/grafana)** · 14 min read · 2 upvotes · 1 comments ## Summary Grafana Labs' Professional Services team shares how they sized, load tested, and now run a production Alloy deployment as a centralized telemetry gateway for an enterprise customer handling ~17M active metric series, 1TB/day of logs, and 1TB/day of traces. Covers resource sizing rules of thumb, HPA configuration (30-100 pods, 70% CPU/90% memory targets), load testing with telemetrygen and k6, key metrics to monitor, and hard-won lessons including WAL memory growth, GOMEMLIMIT as an OOM safeguard, sizing for growth ceilings rather than current baseline, and keeping monitoring out-of-band from the gateway itself. Also outlines future plans for KEDA-based queue-depth autoscaling and per-signal deployment splitting. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does Grafana Alloy run out of memory and get OOM killed under high throughput as a gateway? The write-ahead log (WAL) can grow faster than it drains when Grafana Cloud backpressures under sustained high throughput, eventually causing an out-of-memory kill. Setting GOMEMLIMIT to around 80% of the container memory limit (e.g. 4915MiB for a 6Gi limit) tells the Go garbage collector to reclaim memory aggressively before the hard ceiling is hit, acting as a layered defense alongside HPA CPU and memory thresholds. _Teams tuning Alloy or similar Go-based collectors for production stability can find grounded operational detail like this through daily.dev._ ### How many CPU cores and how much memory does Grafana Alloy need to ingest 17 million active metric series plus terabytes of logs and traces per day? Roughly 195 GB of memory and 28 CPU cores total, broken down as about 187 GB/7 cores for 17M active metric series, 2.1 GB/17.5 cores for 1 TB/day of logs, and 4 GiB/3 cores for 1 TB/day of traces. Pods were sized at 0.5 CPU request and 6Gi memory each, landing around 33-35 pods, with HPA set between 30 and 100 replicas targeting 70% CPU and 90% memory. _Engineers capacity-planning observability pipelines can track real-world sizing benchmarks like this via daily.dev._ ### Should I set a CPU limit on Kubernetes pods running a high-throughput telemetry collector like Alloy? No, only a CPU request should be set, not a CPU limit. Capping CPU artificially creates throttling, one of the most common sources of hidden latency in high-throughput workloads; memory limits combined with autoscaling handle resource pressure more gracefully than a hard CPU cap. _Anyone tuning Kubernetes resource limits for latency-sensitive services can surface practical tradeoffs like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@trevorsuna** · 0 upvotes > Keeping the gateway’s own monitoring out of band is a crucial detail; otherwise the observability path can fail silently with the system it is measuring. Queue depth and per-signal saturation also seem like better scaling inputs than CPU alone once traffic becomes bursty. ## Similar posts on daily.dev - [Kubernetes Monitoring Tools: What Actually Works at Scale](https://daily.dev/posts/kubernetes-monitoring-tools-what-actually-works-at-scale-zdg0bbpod) · Last9 · 2 upvotes · 0 comments - [10 trillion samples a day: Scaling beyond traditional monitoring infra at Databricks](https://daily.dev/posts/10-trillion-samples-a-day-scaling-beyond-traditional-monitoring-infra-at-databricks-nvnrsbnfl) · databricks · 0 upvotes · 0 comments - [See everything, own your data: the Giant Swarm Observability Platform](https://daily.dev/posts/see-everything-own-your-data-the-giant-swarm-observability-platform-apn6qnhua) · Giant Swarm · 0 upvotes · 0 comments - [Native OpenTelemetry inside Alloy: Now you can get the best of both worlds](https://daily.dev/posts/native-opentelemetry-inside-alloy-now-you-can-get-the-best-of-both-worlds-nbjgcmthl) · Grafana Labs · 0 upvotes · 0 comments - [Ensuring reliable OpenTelemetry ingestion at scale](https://daily.dev/posts/ensuring-reliable-opentelemetry-ingestion-at-scale-r8929yipq) · ClickHouse · 3 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#opentelemetry](https://daily.dev/tags/opentelemetry) [View this post on daily.dev](https://daily.dev/posts/how-to-scale-alloy-as-a-central-telemetry-gateway-capacity-planning-load-testing-and-production-l-a0uuqqpox) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How to scale Alloy as a central telemetry gateway: capacity planning, load testing, and production lessons","url":"https://daily.dev/posts/how-to-scale-alloy-as-a-central-telemetry-gateway-capacity-planning-load-testing-and-production-l-a0uuqqpox","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-to-scale-alloy-as-a-central-telemetry-gateway-capacity-planning-load-testing-and-production-l-a0uuqqpox"},"datePublished":"2026-08-24T12:15:05.559Z","dateModified":"2026-09-14T06:18:31.830Z","description":"Grafana Labs' Professional Services team shares how they sized, load tested, and now run a production Alloy deployment as a centralized telemetry gateway for...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/b79ab03f745446107518990c8f169617?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/b79ab03f745446107518990c8f169617?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Grafana Labs","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Grafana Labs","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/769cb8df467e4c428abd75e0c2491d1d","url":"https://daily.dev/sources/grafana"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/how-to-scale-alloy-as-a-central-telemetry-gateway-capacity-planning-load-testing-and-production-l-a0uuqqpox","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"kubernetes,observability,opentelemetry","timeRequired":"PT14M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Grafana Labs","item":"https://daily.dev/sources/grafana"},{"@type":"ListItem","position":3,"name":"How to scale Alloy as a central telemetry gateway: capacity planning, load testing, and production lessons"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/how-to-scale-alloy-as-a-central-telemetry-gateway-capacity-planning-load-testing-and-production-l-a0uuqqpox","comment":[{"@type":"Comment","text":"Keeping the gateway’s own monitoring out of band is a crucial detail; otherwise the observability path can fail silently with the system it is measuring. Queue depth and per-signal saturation also seem like better scaling inputs than CPU alone once traffic becomes bursty.","datePublished":"2026-08-25T14:53:09.138Z","url":"https://daily.dev/posts/a0UUqQpoX#c-ahEHyPRDZ","author":{"@type":"Person","name":"Trevor Suna","url":"https://daily.dev/trevorsuna","image":"https://media.daily.dev/image/upload/s--dZ7gXxpp--/f_auto/v1784081551/avatars/avatar_EMoP47rpuw8DNjhp6R1b6?_a=BAMAMicg0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-to-scale-alloy-as-a-central-telemetry-gateway-capacity-planning-load-testing-and-production-l-a0uuqqpox#faq","mainEntity":[{"@type":"Question","name":"Why does Grafana Alloy run out of memory and get OOM killed under high throughput as a gateway?","acceptedAnswer":{"@type":"Answer","text":"The write-ahead log (WAL) can grow faster than it drains when Grafana Cloud backpressures under sustained high throughput, eventually causing an out-of-memory kill. Setting GOMEMLIMIT to around 80% of the container memory limit (e.g. 4915MiB for a 6Gi limit) tells the Go garbage collector to reclaim memory aggressively before the hard ceiling is hit, acting as a layered defense alongside HPA CPU and memory thresholds. Teams tuning Alloy or similar Go-based collectors for production stability can find grounded operational detail like this through daily.dev."}},{"@type":"Question","name":"How many CPU cores and how much memory does Grafana Alloy need to ingest 17 million active metric series plus terabytes of logs and traces per day?","acceptedAnswer":{"@type":"Answer","text":"Roughly 195 GB of memory and 28 CPU cores total, broken down as about 187 GB/7 cores for 17M active metric series, 2.1 GB/17.5 cores for 1 TB/day of logs, and 4 GiB/3 cores for 1 TB/day of traces. Pods were sized at 0.5 CPU request and 6Gi memory each, landing around 33-35 pods, with HPA set between 30 and 100 replicas targeting 70% CPU and 90% memory. Engineers capacity-planning observability pipelines can track real-world sizing benchmarks like this via daily.dev."}},{"@type":"Question","name":"Should I set a CPU limit on Kubernetes pods running a high-throughput telemetry collector like Alloy?","acceptedAnswer":{"@type":"Answer","text":"No, only a CPU request should be set, not a CPU limit. Capping CPU artificially creates throttling, one of the most common sources of hidden latency in high-throughput workloads; memory limits combined with autoscaling handle resource pressure more gracefully than a hard CPU cap. Anyone tuning Kubernetes resource limits for latency-sensitive services can surface practical tradeoffs like this on daily.dev."}}]} ```