--- title: What Every Engineer Should Know About TCP, Kernel... description: A deep dive into the systems concepts that distinguish senior from staff/principal engineers: kernel vs user space memory boundaries, DMA mechanics, the... canonical: https://daily.dev/posts/what-every-engineer-should-know-about-tcp-kernel-memory-and-load-balancers-3tu8a9dba twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: What Every Engineer Should Know About TCP, Kernel Memory, and Load Balancers | daily.dev og:description: A deep dive into the systems concepts that distinguish senior from staff/principal engineers: kernel vs user space memory boundaries, DMA mechanics, the... og:url: https://daily.dev/posts/what-every-engineer-should-know-about-tcp-kernel-memory-and-load-balancers-3tu8a9dba og:image: https://api.daily.dev/og/posts/3Tu8a9DbA.png og:image:alt: What Every Engineer Should Know About TCP, Kernel Memory, and Load Balancers og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # What Every Engineer Should Know About TCP, Kernel Memory, and Load Balancers **[gitconnected](https://daily.dev/sources/gc)** · 8 min read · 0 upvotes · 0 comments ## Summary A deep dive into the systems concepts that distinguish senior from staff/principal engineers: kernel vs user space memory boundaries, DMA mechanics, the four-copy TCP data path, zero-copy via sendfile() and why TLS breaks it, TCP connection memory and file descriptor costs, and the fundamental architectural difference between L4 and L7 load balancers. L4 LBs use NAT to transparently forward packets while preserving client IPs; L7 LBs must terminate the TCP connection to read HTTP content, making a second connection to the backend unavoidable due to TCP sequence number negotiation. Kafka's high throughput and the existence of X-Forwarded-For headers are explained as direct consequences of these low-level mechanics. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does an L7 load balancer need to open a new TCP connection to the backend instead of just forwarding the client's data? TCP sequence numbers are negotiated during the three-way handshake between two specific endpoints and are not transferable. When a client negotiates a sequence number with the load balancer, the backend never participated in that handshake and has its own sequence number expectations, so forwarding the client's original packets directly would cause the backend to reject them as an invalid connection. A new handshake between the load balancer and backend is unavoidable, which is why L7 load balancing always involves two separate TCP connections. _Understanding why L7 load balancers double up connections helps when debugging latency; daily.dev surfaces deeper networking explainers like this one._ ### Why does read() from a socket always show up as a CPU bottleneck even when the network card has spare capacity? Copying data from the kernel receive buffer to a user-space buffer during a read() call always requires the CPU, unlike the NIC-to-kernel transfer which uses DMA with no CPU involvement. Since user space memory uses virtual addresses that must be translated and permission-checked, DMA cannot write directly there, making this kernel-to-user copy the actual bottleneck in high-throughput benchmarks rather than the NIC's raw transfer capacity. _Engineers chasing throughput bottlenecks can find this kind of kernel-level detail on daily.dev when researching TCP performance._ ### How does Kafka achieve millions of messages per second using zero-copy? Kafka uses the Linux sendfile() system call to move data directly from page cache, which is kernel memory, straight to the network interface card via DMA, without ever copying the data into user space. This eliminates both CPU copies present in the normal data path, leaving only two DMA transfers with zero CPU involvement in data movement, which is what enables such high message throughput. _For engineers evaluating high-throughput messaging systems, daily.dev collects deep dives like this on how Kafka achieves its performance._ ## Similar posts on daily.dev - [Inside the Packet: How Kubernetes Networking Actually Works at L3/L4](https://daily.dev/posts/inside-the-packet-how-kubernetes-networking-actually-works-at-l3-l4-mlktkkrux) · Container Journal · 1 upvotes · 0 comments - [Why application security must start at the load balancer](https://daily.dev/posts/why-application-security-must-start-at-the-load-balancer-bivznty2c) · CSO Online · 0 upvotes · 0 comments - [Medium](https://daily.dev/posts/medium-wddx69mmc) · Medium · 0 upvotes · 0 comments - [EP186: Latency vs. Throughput](https://daily.dev/posts/ep186-latency-vs-throughput-4gkjqtoy1) · ByteByteGo · 10 upvotes · 0 comments --- Tags: [#linux](https://daily.dev/tags/linux), [#networking](https://daily.dev/tags/networking) [View this post on daily.dev](https://daily.dev/posts/what-every-engineer-should-know-about-tcp-kernel-memory-and-load-balancers-3tu8a9dba) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"What Every Engineer Should Know About TCP, Kernel Memory, and Load Balancers","url":"https://daily.dev/posts/what-every-engineer-should-know-about-tcp-kernel-memory-and-load-balancers-3tu8a9dba","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/what-every-engineer-should-know-about-tcp-kernel-memory-and-load-balancers-3tu8a9dba"},"datePublished":"2026-07-23T15:07:40.040Z","dateModified":"2026-09-14T07:03:01.549Z","description":"A deep dive into the systems concepts that distinguish senior from staff/principal engineers: kernel vs user space memory boundaries, DMA mechanics, the...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/656a031da00608c14c9e9425807b70a6?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/656a031da00608c14c9e9425807b70a6?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"gitconnected","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"gitconnected","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/gc","url":"https://daily.dev/sources/gc"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/what-every-engineer-should-know-about-tcp-kernel-memory-and-load-balancers-3tu8a9dba","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"linux,networking","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"gitconnected","item":"https://daily.dev/sources/gc"},{"@type":"ListItem","position":3,"name":"What Every Engineer Should Know About TCP, Kernel Memory, and Load Balancers"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/what-every-engineer-should-know-about-tcp-kernel-memory-and-load-balancers-3tu8a9dba#faq","mainEntity":[{"@type":"Question","name":"Why does an L7 load balancer need to open a new TCP connection to the backend instead of just forwarding the client's data?","acceptedAnswer":{"@type":"Answer","text":"TCP sequence numbers are negotiated during the three-way handshake between two specific endpoints and are not transferable. When a client negotiates a sequence number with the load balancer, the backend never participated in that handshake and has its own sequence number expectations, so forwarding the client's original packets directly would cause the backend to reject them as an invalid connection. A new handshake between the load balancer and backend is unavoidable, which is why L7 load balancing always involves two separate TCP connections. Understanding why L7 load balancers double up connections helps when debugging latency; daily.dev surfaces deeper networking explainers like this one."}},{"@type":"Question","name":"Why does read() from a socket always show up as a CPU bottleneck even when the network card has spare capacity?","acceptedAnswer":{"@type":"Answer","text":"Copying data from the kernel receive buffer to a user-space buffer during a read() call always requires the CPU, unlike the NIC-to-kernel transfer which uses DMA with no CPU involvement. Since user space memory uses virtual addresses that must be translated and permission-checked, DMA cannot write directly there, making this kernel-to-user copy the actual bottleneck in high-throughput benchmarks rather than the NIC's raw transfer capacity. Engineers chasing throughput bottlenecks can find this kind of kernel-level detail on daily.dev when researching TCP performance."}},{"@type":"Question","name":"How does Kafka achieve millions of messages per second using zero-copy?","acceptedAnswer":{"@type":"Answer","text":"Kafka uses the Linux sendfile() system call to move data directly from page cache, which is kernel memory, straight to the network interface card via DMA, without ever copying the data into user space. This eliminates both CPU copies present in the normal data path, leaving only two DMA transfers with zero CPU involvement in data movement, which is what enables such high message throughput. For engineers evaluating high-throughput messaging systems, daily.dev collects deep dives like this on how Kafka achieves its performance."}}]} ```