Oxide Computer Company details how they tuned network performance across every layer of their self-built rack stack. The post covers jumbo frames (MTU 8500 for guests vs. 9000 underlay), explaining why external traffic saw a ~4x throughput improvement while internal VPC traffic barely changed. It also covers TCP segmentation offloads, CPU placement for virtual NIC worker threads, flow parallelism via ECMP, and IPv6 as a first-class citizen with parity to IPv4 throughput. Key engineering decisions include why MTU belongs on the instance rather than the switch port, the headroom reasoning behind 8500 bytes, and the operational risks of jumbo frames when ICMP path-MTU discovery signals get dropped. The broader theme is that owning the full stack — from physical fabric to guest NIC — enables coordinated tuning no single-vendor solution can match.
Table of contents
Why 8500 and not 9000What about a per-port knob?What the setting buys, and the fabric under itJumbo frames are not a go-fast buttonThe one rule for jumbo frames191K Impressions2 Comments