--- title: The Hidden Cost of “Just Works” Load Balancing in a... description: Multi-AZ Kubernetes clusters running Istio default to random load balancing across availability zones, which quietly adds latency and AWS cross-AZ data... canonical: https://daily.dev/posts/the-hidden-cost-of-just-works-load-balancing-in-a-service-mesh-yohlgudu2 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The Hidden Cost of “Just Works” Load Balancing in a Service Mesh | daily.dev og:description: Multi-AZ Kubernetes clusters running Istio default to random load balancing across availability zones, which quietly adds latency and AWS cross-AZ data... og:url: https://daily.dev/posts/the-hidden-cost-of-just-works-load-balancing-in-a-service-mesh-yohlgudu2 og:image: https://api.daily.dev/og/posts/YoHLgUDu2.png og:image:alt: The Hidden Cost of “Just Works” Load Balancing in a Service Mesh og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The Hidden Cost of “Just Works” Load Balancing in a Service Mesh **[Container Journal](https://daily.dev/sources/container_journal)** · 5 min read · 1 upvotes · 0 comments ## Summary Multi-AZ Kubernetes clusters running Istio default to random load balancing across availability zones, which quietly adds latency and AWS cross-AZ data transfer costs. Tracing data from a mid-sized cluster running ~3,500 RPS showed same-AZ p50 latency of 15-18ms versus 25-30ms cross-AZ, and cross-AZ traffic charges around $600/month before optimization. The fix is Istio's locality-aware load balancing via DestinationRule, using an 80/10/10 weighted split rather than 100% local-only routing, combined with even pod distribution across AZs, spread ingress gateways, and disabling NLB cross-zone load balancing. The tradeoff is added operational complexity in reasoning about traffic distribution. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I configure Istio locality-aware load balancing to reduce cross-AZ traffic in a multi-AZ Kubernetes cluster? Use an Istio DestinationRule with localityLbSetting enabled and a distribute block sending 80% of traffic to the local AZ and 10% each to the other two AZs, combined with outlier detection (consecutiveErrors: 5, interval: 30s, baseEjectionTime: 30s, maxEjectionPercent: 50). Avoid routing 100% locally since it removes the ability to absorb a bad deploy or hot pod and complicates full AZ failure recovery. _daily.dev surfaces practical service mesh configuration patterns for engineers tuning multi-AZ traffic costs._ ### How much does cross-AZ traffic cost on AWS and how does it affect latency in a service mesh? AWS charges $0.01/GB for traffic crossing availability zones within a region, and in one mid-sized production cluster running about 3,500 RPS, this added up to roughly $600/month before any fix, driven largely by internal service-to-service, database, and Kafka traffic. Cross-AZ requests also ran 25-30ms p50 versus 15-18ms p50 for same-AZ requests, a 40-65% latency increase. _Engineers debugging unexpected AWS data transfer bills can track these cost-optimization patterns on daily.dev._ ### What settings are required to make Istio locality-aware load balancing actually work as intended? Three conditions must hold: pods must be evenly distributed across availability zones using topologySpreadConstraints with maxSkew 1, ingress gateway pods must also be spread across all AZs rather than concentrated in one, and NLB cross-zone load balancing must be disabled via the aws-load-balancer-cross-zone-load-balancing-enabled annotation set to false. Skipping any one reintroduces the imbalance the locality policy was meant to fix. _daily.dev helps infrastructure engineers keep up with the operational details behind mesh networking tradeoffs._ ## Similar posts on daily.dev - [Cross-AZ Traffic on EKS: The Cost Nobody Watches](https://daily.dev/posts/cross-az-traffic-on-eks-the-cost-nobody-watches-hylxtwqqh) · AWS Fundamentals · 7 upvotes · 0 comments - [Client-Side Load Balancing at a Million Requests Per Second](https://daily.dev/posts/client-side-load-balancing-at-a-million-requests-per-second-bvi2j48ww) · Zalando · 20 upvotes · 1 comments - [How Zalando Built an In-Process Client-Side Load Balancer for One Million Requests per Second](https://daily.dev/posts/how-zalando-built-an-in-process-client-side-load-balancer-for-one-million-requests-per-second-gwprc272c) · InfoQ · 0 upvotes · 0 comments - [The 7 Kubernetes Cost Drivers Most Teams Miss](https://daily.dev/posts/the-7-kubernetes-cost-drivers-most-teams-miss-qntoy99sp) · Cast AI · 0 upvotes · 0 comments - [Multi AZ by Default, Local by Design, Cutting Cross Zone Traffic With Cilium](https://daily.dev/posts/multi-az-by-default-local-by-design-cutting-cross-zone-traffic-with-cilium-can44cqgu) · Isovalent · 1 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#kubernetes](https://daily.dev/tags/kubernetes), [#istio](https://daily.dev/tags/istio), [#service-mesh](https://daily.dev/tags/service-mesh) [View this post on daily.dev](https://daily.dev/posts/the-hidden-cost-of-just-works-load-balancing-in-a-service-mesh-yohlgudu2) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The Hidden Cost of “Just Works” Load Balancing in a Service Mesh","url":"https://daily.dev/posts/the-hidden-cost-of-just-works-load-balancing-in-a-service-mesh-yohlgudu2","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-hidden-cost-of-just-works-load-balancing-in-a-service-mesh-yohlgudu2"},"datePublished":"2026-08-14T22:48:41.219Z","dateModified":"2026-09-14T06:54:26.714Z","description":"Multi-AZ Kubernetes clusters running Istio default to random load balancing across availability zones, which quietly adds latency and AWS cross-AZ data...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c07e6ea4c40351618c39c07d80f2e4e9?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c07e6ea4c40351618c39c07d80f2e4e9?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Container Journal","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Container Journal","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/45be8dbd2bbe4dd6ac35de519c6e4cd2","url":"https://daily.dev/sources/container_journal"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-hidden-cost-of-just-works-load-balancing-in-a-service-mesh-yohlgudu2","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,kubernetes,istio,service-mesh","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Container Journal","item":"https://daily.dev/sources/container_journal"},{"@type":"ListItem","position":3,"name":"The Hidden Cost of “Just Works” Load Balancing in a Service Mesh"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-hidden-cost-of-just-works-load-balancing-in-a-service-mesh-yohlgudu2#faq","mainEntity":[{"@type":"Question","name":"How do I configure Istio locality-aware load balancing to reduce cross-AZ traffic in a multi-AZ Kubernetes cluster?","acceptedAnswer":{"@type":"Answer","text":"Use an Istio DestinationRule with localityLbSetting enabled and a distribute block sending 80% of traffic to the local AZ and 10% each to the other two AZs, combined with outlier detection (consecutiveErrors: 5, interval: 30s, baseEjectionTime: 30s, maxEjectionPercent: 50). Avoid routing 100% locally since it removes the ability to absorb a bad deploy or hot pod and complicates full AZ failure recovery. daily.dev surfaces practical service mesh configuration patterns for engineers tuning multi-AZ traffic costs."}},{"@type":"Question","name":"How much does cross-AZ traffic cost on AWS and how does it affect latency in a service mesh?","acceptedAnswer":{"@type":"Answer","text":"AWS charges $0.01/GB for traffic crossing availability zones within a region, and in one mid-sized production cluster running about 3,500 RPS, this added up to roughly $600/month before any fix, driven largely by internal service-to-service, database, and Kafka traffic. Cross-AZ requests also ran 25-30ms p50 versus 15-18ms p50 for same-AZ requests, a 40-65% latency increase. Engineers debugging unexpected AWS data transfer bills can track these cost-optimization patterns on daily.dev."}},{"@type":"Question","name":"What settings are required to make Istio locality-aware load balancing actually work as intended?","acceptedAnswer":{"@type":"Answer","text":"Three conditions must hold: pods must be evenly distributed across availability zones using topologySpreadConstraints with maxSkew 1, ingress gateway pods must also be spread across all AZs rather than concentrated in one, and NLB cross-zone load balancing must be disabled via the aws-load-balancer-cross-zone-load-balancing-enabled annotation set to false. Skipping any one reintroduces the imbalance the locality policy was meant to fix. daily.dev helps infrastructure engineers keep up with the operational details behind mesh networking tradeoffs."}}]} ```