--- title: "How to scale Alloy as a central telemetry gateway: capacity planning, load testing, and production lessons" url: https://daily.dev/posts/how-to-scale-alloy-as-a-central-telemetry-gateway-capacity-planning-load-testing-and-production-l-a0uuqqpox source_url: https://grafana.com/blog/how-to-scale-alloy-as-a-central-telemetry-gateway-capacity-planning-load-testing-and-production-lessons type: article source: "Grafana Labs" published: 2026-08-24T12:15:05.559Z updated: 2026-08-24T12:15:30.485Z tags: ["kubernetes", "observability", "opentelemetry"] reading_time: 14 upvotes: 2 comments: 1 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How to scale Alloy as a central telemetry gateway: capacity planning, load testing, and production lessons **[Grafana Labs](https://daily.dev/sources/grafana)** · 14 min read · 2 upvotes · 1 comments ## Summary Grafana Labs' Professional Services team shares how they sized, load tested, and now run a production Alloy deployment as a centralized telemetry gateway for an enterprise customer handling ~17M active metric series, 1TB/day of logs, and 1TB/day of traces. Covers resource sizing rules of thumb, HPA configuration (30-100 pods, 70% CPU/90% memory targets), load testing with telemetrygen and k6, key metrics to monitor, and hard-won lessons including WAL memory growth, GOMEMLIMIT as an OOM safeguard, sizing for growth ceilings rather than current baseline, and keeping monitoring out-of-band from the gateway itself. Also outlines future plans for KEDA-based queue-depth autoscaling and per-signal deployment splitting. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do you prevent Grafana Alloy's write-ahead log from causing an OOM kill under high throughput? Set GOMEMLIMIT to roughly 80% of the container's memory limit so Go's garbage collector aggressively reclaims memory before the kernel kills the pod. This acts as a soft-limit trigger: when GOMEMLIMIT is hit, CPU rises, which triggers the HPA's 70% CPU threshold to scale out more pods before memory hits the 90% HPA safety net or the hard pod memory limit causes an OOMKill. _daily.dev surfaces production lessons like this for engineers tuning memory-hungry telemetry pipelines._ ### How many CPU cores and how much memory does Grafana Alloy need to ingest 17M active metric series plus 1TB/day of logs and traces? Roughly 195GB of memory and 28 CPU cores total: about 187GB/7 cores for 17M active metric series, 2.1GB/17.5 cores for 1TB/day of logs peaking at 17.5MB/s, and 4GiB/3 cores for 1TB/day of traces peaking at 23MB/s. Pods were sized at 6GiB memory with 0.5 CPU requests and no CPU limit, landing around 33-35 pods under normal load. _developers planning telemetry infrastructure capacity can find sizing benchmarks like this on daily.dev._ ### Should I set a CPU limit on Kubernetes pods running high-throughput workloads like an OTLP collector? No, only a CPU request should be set, with no CPU limit, because CPU throttling in Kubernetes is one of the most common sources of hidden latency in high-throughput workloads. Capping CPU artificially creates pressure that memory limits combined with horizontal pod autoscaling handle more gracefully, since autoscaling can add capacity rather than throttle existing pods. _engineers weighing Kubernetes resource limit tradeoffs track guidance like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@trevorsuna** · 0 upvotes > Keeping the gateway’s own monitoring out of band is a crucial detail; otherwise the observability path can fail silently with the system it is measuring. Queue depth and per-signal saturation also seem like better scaling inputs than CPU alone once traffic becomes bursty. ## Similar posts on daily.dev - [Kubernetes Monitoring Tools: What Actually Works at Scale](https://daily.dev/posts/kubernetes-monitoring-tools-what-actually-works-at-scale-zdg0bbpod) · Last9 · 2 upvotes · 0 comments - [10 trillion samples a day: Scaling beyond traditional monitoring infra at Databricks](https://daily.dev/posts/10-trillion-samples-a-day-scaling-beyond-traditional-monitoring-infra-at-databricks-nvnrsbnfl) · databricks · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#observability](https://daily.dev/tags/observability), [#opentelemetry](https://daily.dev/tags/opentelemetry) [View this post on daily.dev](https://daily.dev/posts/how-to-scale-alloy-as-a-central-telemetry-gateway-capacity-planning-load-testing-and-production-l-a0uuqqpox)