Google and Anyscale have jointly engineered major performance improvements to Ray Serve LLM on GKE, delivering up to 5x higher throughput and 8x lower latency compared to previous configurations. Three key architectural changes drive these gains: HAProxy integration for internal request routing to reduce Python runtime saturation, a direct token streaming architecture that bypasses the ingress router for streaming data, and a v2 Ray executor backend for vLLM that moves Ray out of the data plane for asynchronous scheduling. Benchmarks were run on GKE clusters with NVIDIA HGX B200 GPUs using Gemma 4 E2B, showing performance now comparable to native vLLM while retaining Ray's ecosystem flexibility. The improvements are available in Ray 2.56 and later.
