--- title: The three-layer GKE network mystery | daily.dev description: A GKE outage that blocked around 800 developers from deploying for weeks turned out to involve three separate bugs stacked across the networking stack. A USPS... canonical: https://daily.dev/posts/the-three-layer-gke-network-mystery-4cc8dwtvl twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The three-layer GKE network mystery | daily.dev og:description: A GKE outage that blocked around 800 developers from deploying for weeks turned out to involve three separate bugs stacked across the networking stack. A USPS... og:url: https://daily.dev/posts/the-three-layer-gke-network-mystery-4cc8dwtvl og:image: https://api.daily.dev/og/posts/4CC8DwTVl.png og:image:alt: The three-layer GKE network mystery og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The three-layer GKE network mystery **[Google Cloud](https://daily.dev/sources/gcp)** · 19 min read · 0 upvotes · 0 comments ## Summary A GKE outage that blocked around 800 developers from deploying for weeks turned out to involve three separate bugs stacked across the networking stack. A USPS update started silently dropping RFC 5961 'challenge ACK' packets used by Private Service Connect, packet reordering at the PIGv virtual NIC caused client FIN packets to arrive out of order, and a Linux kernel 6.1 bug in tcp_ack() silently discarded valid-but-old ACKs during connection close. Combined with Envoy's half-close (SHUT_WR) keeping sockets non-orphaned, connections got stuck in FIN_WAIT_2 indefinitely, their ports got reused by PSC's NAT, and new connections timed out. Google rolled back the USPS change, fixed the PSC proxy protocol handling, and shipped a kernel patch (upstream commit 795a7dfbc3d9) in GKE 1.33.5. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why would GKE pods intermittently time out connecting to the Kubernetes API server behind Private Service Connect? Three bugs combined: a Private Service Connect NAT layer update started dropping RFC 5961 'challenge ACK' packets, a virtual NIC reordering issue delayed client FIN packets, and a Linux kernel 6.1 bug silently discarded valid old ACKs during connection close. Together these left sockets stuck in FIN_WAIT_2 indefinitely, so reused source ports triggered dropped challenge ACKs and i/o timeouts. _Debugging stubborn control-plane timeouts across network layers is easier when you've seen a breakdown like this on daily.dev._ ### What Linux kernel bug causes sockets to stay stuck in FIN_WAIT_2 forever? In kernel 6.1, tcp_ack() could return zero for an ACK that was old but still acceptable under RFC 5961, and the FIN processing path treated that return value as 'not processed,' silently discarding a valid received FIN. This kept sockets stuck in FIN_WAIT_2 indefinitely instead of transitioning to TIME_WAIT. It was fixed upstream in commit 795a7dfbc3d9 by Eric Dumazet, merged into Linux 6.9.7 and backported to the 6.6.y and 6.1.y LTS branches. _Tracking kernel networking fixes like this on daily.dev helps you decide when a backport actually matters for your stack._ ### Why does Envoy's shutdown(SHUT_WR) half-close keep a TCP socket from timing out in FIN_WAIT_2? The tcp_fin_timeout kernel parameter only forces closure of orphaned sockets, those no process still holds open. Envoy's half-close via shutdown(SHUT_WR) closes only the write side while keeping the socket open to still receive data, so the kernel never considers it orphaned and tcp_fin_timeout never starts, letting the socket sit in FIN_WAIT_2 indefinitely. _Understanding half-close behavior like this helps when you're choosing a proxy's connection-termination strategy; developers compare these tradeoffs on daily.dev._ ## Similar posts on daily.dev - [April 2026 Outage Post-Mortem - Jim's Pckt](https://daily.dev/posts/april-2026-outage-post-mortem---jim-s-pckt-rxhhl4gj7) · Hacker News · 0 upvotes · 0 comments - [Thoughts on the Bluesky public incident write-up](https://daily.dev/posts/thoughts-on-the-bluesky-public-incident-write-up-7m6kk8i9e) · Surfing Complexity · 32 upvotes · 1 comments - [How your cloud could get hacked: an analysis of a real-world vulnerability chain](https://daily.dev/posts/how-your-cloud-could-get-hacked-an-analysis-of-a-real-world-vulnerability-chain-fpulkynl5) · Securelist · 0 upvotes · 0 comments --- Tags: [#linux](https://daily.dev/tags/linux), [#kubernetes](https://daily.dev/tags/kubernetes), [#networking](https://daily.dev/tags/networking), [#gke](https://daily.dev/tags/gke) [View this post on daily.dev](https://daily.dev/posts/the-three-layer-gke-network-mystery-4cc8dwtvl) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The three-layer GKE network mystery","url":"https://daily.dev/posts/the-three-layer-gke-network-mystery-4cc8dwtvl","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-three-layer-gke-network-mystery-4cc8dwtvl"},"datePublished":"2026-10-30T20:53:27.610Z","dateModified":"2026-10-30T20:54:01.828Z","description":"A GKE outage that blocked around 800 developers from deploying for weeks turned out to involve three separate bugs stacked across the networking stack. A USPS...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/78840f1087de90cbe3e45f1e24ca3e3d?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/78840f1087de90cbe3e45f1e24ca3e3d?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Google Cloud","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Google Cloud","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/gcp","url":"https://daily.dev/sources/gcp"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-three-layer-gke-network-mystery-4cc8dwtvl","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"linux,kubernetes,networking,gke","timeRequired":"PT19M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Google Cloud","item":"https://daily.dev/sources/gcp"},{"@type":"ListItem","position":3,"name":"The three-layer GKE network mystery"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-three-layer-gke-network-mystery-4cc8dwtvl#faq","mainEntity":[{"@type":"Question","name":"Why would GKE pods intermittently time out connecting to the Kubernetes API server behind Private Service Connect?","acceptedAnswer":{"@type":"Answer","text":"Three bugs combined: a Private Service Connect NAT layer update started dropping RFC 5961 'challenge ACK' packets, a virtual NIC reordering issue delayed client FIN packets, and a Linux kernel 6.1 bug silently discarded valid old ACKs during connection close. Together these left sockets stuck in FIN_WAIT_2 indefinitely, so reused source ports triggered dropped challenge ACKs and i/o timeouts. Debugging stubborn control-plane timeouts across network layers is easier when you've seen a breakdown like this on daily.dev."}},{"@type":"Question","name":"What Linux kernel bug causes sockets to stay stuck in FIN_WAIT_2 forever?","acceptedAnswer":{"@type":"Answer","text":"In kernel 6.1, tcp_ack() could return zero for an ACK that was old but still acceptable under RFC 5961, and the FIN processing path treated that return value as 'not processed,' silently discarding a valid received FIN. This kept sockets stuck in FIN_WAIT_2 indefinitely instead of transitioning to TIME_WAIT. It was fixed upstream in commit 795a7dfbc3d9 by Eric Dumazet, merged into Linux 6.9.7 and backported to the 6.6.y and 6.1.y LTS branches. Tracking kernel networking fixes like this on daily.dev helps you decide when a backport actually matters for your stack."}},{"@type":"Question","name":"Why does Envoy's shutdown(SHUT_WR) half-close keep a TCP socket from timing out in FIN_WAIT_2?","acceptedAnswer":{"@type":"Answer","text":"The tcp_fin_timeout kernel parameter only forces closure of orphaned sockets, those no process still holds open. Envoy's half-close via shutdown(SHUT_WR) closes only the write side while keeping the socket open to still receive data, so the kernel never considers it orphaned and tcp_fin_timeout never starts, letting the socket sit in FIN_WAIT_2 indefinitely. Understanding half-close behavior like this helps when you're choosing a proxy's connection-termination strategy; developers compare these tradeoffs on daily.dev."}}]} ```