The Hidden Cost of “Just Works” Load Balancing in a Service Mesh
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Multi-AZ Kubernetes clusters running Istio default to random load balancing across availability zones, which quietly adds latency and AWS cross-AZ data transfer costs. Tracing data from a mid-sized cluster running ~3,500 RPS showed same-AZ p50 latency of 15-18ms versus 25-30ms cross-AZ, and cross-AZ traffic charges around $600/month before optimization. The fix is Istio's locality-aware load balancing via DestinationRule, using an 80/10/10 weighted split rather than 100% local-only routing, combined with even pod distribution across AZs, spread ingress gateways, and disabling NLB cross-zone load balancing. The tradeoff is added operational complexity in reasoning about traffic distribution.
Table of contents
The Default Nobody ConfiguresWhat It Actually CostsThe Fix: Locality-Aware, Not Locality-OnlyWhat You Get, and What You Give UpBefore You Roll This OutRelatedQuestions this post answers
How do I configure Istio DestinationRule for locality-aware load balancing across availability zones?
Set localityLbSetting.enabled to true in the DestinationRule's trafficPolicy.loadBalancer, then use the distribute field to specify weighted percentages per zone, such as 80% to the local AZ and 10% each to the other two. Pair this with outlierDetection settings like consecutiveErrors: 5, interval: 30s, and baseEjectionTime: 30s so unhealthy endpoints are automatically ejected. daily.dev surfaces practical Istio traffic policy configs for engineers tuning multi-AZ mesh setups.
Why is my cross-AZ traffic in a multi-AZ Kubernetes cluster with Istio so expensive and slow?
Kubernetes Services and Istio's Envoy sidecars distribute traffic randomly across all healthy endpoints with no zone awareness, so a pod has roughly a two-in-three chance of calling a downstream pod in a different AZ across three zones. In one mid-sized cluster running about 3,500 RPS, this pushed weighted p50 latency to around 24ms instead of 17ms and generated roughly $600 per month in AWS cross-AZ data transfer charges at $0.01/GB. Engineers debugging mesh latency and AWS bandwidth bills can track fixes like this on daily.dev.
Should I route 100% of Istio traffic to the local availability zone for cost savings?
No, routing all traffic locally removes the ability to absorb a bad deploy or a hot pod in that zone and makes recovery from a full AZ failure much harder. A better pattern is a weighted split, such as 80% same-AZ traffic with 10% each to the other two zones, combined with outlier detection so unhealthy endpoints get ejected automatically. daily.dev helps infrastructure teams weigh tradeoffs like locality routing before rolling changes to production.