--- title: Performance Has Layers | daily.dev description: Oxide Computer Company details how they tuned network performance across every layer of their self-built rack stack. The post covers jumbo frames (MTU 8500 for... canonical: https://daily.dev/posts/performance-has-layers-ega2gcrnh twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Performance Has Layers | daily.dev og:description: Oxide Computer Company details how they tuned network performance across every layer of their self-built rack stack. The post covers jumbo frames (MTU 8500 for... og:url: https://daily.dev/posts/performance-has-layers-ega2gcrnh og:image: https://api.daily.dev/og/posts/egA2gCRnH.png og:image:alt: Performance Has Layers og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Performance Has Layers **[Oxide](https://daily.dev/sources/oxide)** · 14 min read · 97 upvotes · 2 comments ## Summary Oxide Computer Company details how they tuned network performance across every layer of their self-built rack stack. The post covers jumbo frames (MTU 8500 for guests vs. 9000 underlay), explaining why external traffic saw a ~4x throughput improvement while internal VPC traffic barely changed. It also covers TCP segmentation offloads, CPU placement for virtual NIC worker threads, flow parallelism via ECMP, and IPv6 as a first-class citizen with parity to IPv4 throughput. Key engineering decisions include why MTU belongs on the instance rather than the switch port, the headroom reasoning behind 8500 bytes, and the operational risks of jumbo frames when ICMP path-MTU discovery signals get dropped. The broader theme is that owning the full stack — from physical fabric to guest NIC — enables coordinated tuning no single-vendor solution can match. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did Oxide choose an 8500-byte MTU for external jumbo frames instead of the full 9000-byte underlay? An 8500-byte guest MTU leaves headroom above the roughly 70-78 bytes of Geneve/IPv6/UDP encapsulation overhead on a 9000-byte underlay. Oxide reserves about 500 bytes for future Geneve features and routing protocol extension headers, and keeps the encapsulation stack shallow enough that switch hardware, which often parses only the first 256-512 bytes of a packet, can still read the fields needed for forwarding decisions. _Anyone tuning jumbo frame MTUs can compare real-world sizing tradeoffs like this one on daily.dev._ ### How much does enabling jumbo frames improve external network throughput compared to internal VPC traffic? External throughput improved roughly 4x, rising from about 7.7-8 Gbps to 32.7-34.2 Gbps per pair on a 40G uplink when moving from a 1500-byte to an 8500-byte MTU. Internal VPC traffic barely changed (about 6-7% improvement) since it already used large frames internally, closing the internal-versus-external throughput gap from roughly 7x down to about 1.7x. _Engineers evaluating jumbo frame gains for their own network paths can track benchmarks like these on daily.dev._ ### What causes a silent network black hole when enabling jumbo frames without checking the full path MTU? When a packet is too large for a link along its path and the resulting ICMP fragmentation-needed (or IPv6 packet-too-big) message gets dropped by a firewall, router policy, or overloaded hop, the sender never learns to lower its frame size. The TCP handshake completes but the application then stalls and transfers nothing, and this failure mode is worse for IPv6 since ICMPv6 handling is less reliable. _Teams rolling out jumbo frames can avoid this exact failure mode by following path-MTU audit practices shared on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@bits\_and\_bytes** · 0 upvotes > Great write-up. The MTU sizing rationale and the focus on end-to-end tuning across the entire stack were particularly insightful. Impressive engineering work. ## Similar posts on daily.dev - [Fornax's Guide To Ridiculously Fast Ethernet](https://daily.dev/posts/fornax-s-guide-to-ridiculously-fast-ethernet-azd9fuiz0) · Programming Digest · 12 upvotes · 0 comments --- Tags: [#infrastructure](https://daily.dev/tags/infrastructure) [View this post on daily.dev](https://daily.dev/posts/performance-has-layers-ega2gcrnh) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Performance Has Layers","url":"https://daily.dev/posts/performance-has-layers-ega2gcrnh","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/performance-has-layers-ega2gcrnh"},"datePublished":"2026-06-18T16:23:44.281Z","dateModified":"2026-06-19T04:20:23.742Z","description":"Oxide Computer Company details how they tuned network performance across every layer of their self-built rack stack. The post covers jumbo frames (MTU 8500 for...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8024a48458e6ea30886021eea5d8ec6a?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8024a48458e6ea30886021eea5d8ec6a?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Oxide","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Oxide","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/a06afd0e0fbf40ec9d3128cd1c103021","url":"https://daily.dev/sources/oxide"},"commentCount":2,"discussionUrl":"https://daily.dev/posts/performance-has-layers-ega2gcrnh","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":97},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":2}],"keywords":"infrastructure","timeRequired":"PT14M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Oxide","item":"https://daily.dev/sources/oxide"},{"@type":"ListItem","position":3,"name":"Performance Has Layers"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/performance-has-layers-ega2gcrnh","comment":[{"@type":"Comment","text":"Great write-up. The MTU sizing rationale and the focus on end-to-end tuning across the entire stack were particularly insightful. Impressive engineering work.","datePublished":"2026-06-21T02:40:03.674Z","url":"https://daily.dev/posts/egA2gCRnH#c-EJwyl0imx","author":{"@type":"Person","name":"bits_007","url":"https://daily.dev/bits_and_bytes","image":"https://avatars.githubusercontent.com/u/182549945?v=4"}}]} ```