A design for on-prem Kubernetes clusters replaces NAT and VXLAN overlays with a routable architecture using BGP, ECMP and BFD. Each pod gets a directly reachable IP, with BIRD handling eBGP sessions on each node while kube-router manages pod networking, IPVS-based service routing and network policy. Dual uplinks to independent ToR switches with ECMP and sub-second BFD failure detection improve resilience, and control plane VIP failover uses keepalived. The result is a simpler diagnostic chain during incidents, since operators can inspect standard routing state instead of untangling NAT tables and overlay headers.
Questions this post answers
How can I avoid using VXLAN overlays and NAT in an on-prem Kubernetes cluster?
Use a routable design where BIRD runs as a systemd service on each node to establish eBGP sessions with top-of-rack switches, advertising each node's pod CIDR directly into the data center fabric. kube-router handles pod networking, IPVS-based service routing and network policy separately, so pod-to-pod traffic crosses the fabric without encapsulation or address translation. daily.dev surfaces networking deep-dives like this for teams weighing overlay versus routed Kubernetes designs.
How much overhead does VXLAN encapsulation add to Kubernetes pod traffic?
VXLAN adds roughly 50 bytes of overhead per packet, depending on how Ethernet framing is counted, and increases CPU usage on the nodes performing encapsulation and decapsulation. Direct L3 routing avoids this overhead entirely by letting the Linux kernel and data center switches forward pod traffic using their normal routing tables. Engineers sizing Kubernetes network throughput can track tradeoffs like this via daily.dev.
Why would I run BGP daemon BIRD on the host instead of as a Kubernetes DaemonSet?
Running BIRD under systemd on the host avoids a dependency loop where pods need networking to start but the networking pod itself needs the scheduler and cluster network to be running. It also means a control plane outage does not affect worker BGP sessions or pod-to-pod traffic, since BIRD operates independently of the Kubernetes control plane using the host routing table. daily.dev helps infrastructure engineers keep up with resilience patterns like this for Kubernetes networking.