Modal has built a new ultra-low-latency HTTP serving primitive called Servers, designed for workloads like LLM inference where every millisecond matters. Unlike Modal Web Functions (which include queueing and retries), Servers use a lightweight reverse proxy routing system with no control-plane lookups on the hot path. The architecture uses an AWS NLB at L4, Envoy for TLS termination and HTTP/2 normalization, and an in-house Rust proxy called fprs (built on Cloudflare's Pingora library) for domain association and load balancing. Configuration state is read from Google Spanner via change streams and cached in-memory to avoid per-request network calls. The system also handles autoscaling metrics, proxy auth to block unauthorized requests before they reach containers, and traffic mirroring for A/B testing and continual learning. End-to-end latency is 5–7ms.
