Grafana Labs' Professional Services team shares how they sized, load tested, and now run a production Alloy deployment as a centralized telemetry gateway for an enterprise customer handling ~17M active metric series, 1TB/day of logs, and 1TB/day of traces. Covers resource sizing rules of thumb, HPA configuration (30-100 pods, 70% CPU/90% memory targets), load testing with telemetrygen and k6, key metrics to monitor, and hard-won lessons including WAL memory growth, GOMEMLIMIT as an OOM safeguard, sizing for growth ceilings rather than current baseline, and keeping monitoring out-of-band from the gateway itself. Also outlines future plans for KEDA-based queue-depth autoscaling and per-signal deployment splitting.
Questions this post answers
How do you prevent Grafana Alloy's write-ahead log from causing an OOM kill under high throughput?
Set GOMEMLIMIT to roughly 80% of the container's memory limit so Go's garbage collector aggressively reclaims memory before the kernel kills the pod. This acts as a soft-limit trigger: when GOMEMLIMIT is hit, CPU rises, which triggers the HPA's 70% CPU threshold to scale out more pods before memory hits the 90% HPA safety net or the hard pod memory limit causes an OOMKill. daily.dev surfaces production lessons like this for engineers tuning memory-hungry telemetry pipelines.
How many CPU cores and how much memory does Grafana Alloy need to ingest 17M active metric series plus 1TB/day of logs and traces?
Roughly 195GB of memory and 28 CPU cores total: about 187GB/7 cores for 17M active metric series, 2.1GB/17.5 cores for 1TB/day of logs peaking at 17.5MB/s, and 4GiB/3 cores for 1TB/day of traces peaking at 23MB/s. Pods were sized at 6GiB memory with 0.5 CPU requests and no CPU limit, landing around 33-35 pods under normal load. developers planning telemetry infrastructure capacity can find sizing benchmarks like this on daily.dev.
Should I set a CPU limit on Kubernetes pods running high-throughput workloads like an OTLP collector?
No, only a CPU request should be set, with no CPU limit, because CPU throttling in Kubernetes is one of the most common sources of hidden latency in high-throughput workloads. Capping CPU artificially creates pressure that memory limits combined with horizontal pod autoscaling handle more gracefully, since autoscaling can add capacity rather than throttle existing pods. engineers weighing Kubernetes resource limit tradeoffs track guidance like this on daily.dev.
1.7K Impressions1 Comment