--- title: "Improving Ray Serve LLM on GKE throughput, latency" url: https://daily.dev/posts/improving-ray-serve-llm-on-gke-throughput-latency-el4asvvua source_url: https://cloud.google.com/blog/products/containers-kubernetes/improving-ray-serve-llm-on-gke-throughput-latency type: article source: "Google Cloud" published: 2026-06-18T16:03:22.716Z updated: 2026-06-18T17:47:01.505Z tags: ["gcp", "ai-inference", "vllm"] reading_time: 3 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Improving Ray Serve LLM on GKE throughput, latency **[Google Cloud](https://daily.dev/sources/gcp)** · 3 min read · 0 upvotes · 0 comments ## Summary Google and Anyscale have jointly engineered major performance improvements to Ray Serve LLM on GKE, delivering up to 5x higher throughput and 8x lower latency compared to previous configurations. Three key architectural changes drive these gains: HAProxy integration for internal request routing to reduce Python runtime saturation, a direct token streaming architecture that bypasses the ingress router for streaming data, and a v2 Ray executor backend for vLLM that moves Ray out of the data plane for asynchronous scheduling. Benchmarks were run on GKE clusters with NVIDIA HGX B200 GPUs using Gemma 4 E2B, showing performance now comparable to native vLLM while retaining Ray's ecosystem flexibility. The improvements are available in Ray 2.56 and later. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Ray on GKE: New features for AI scheduling and scaling](https://daily.dev/posts/ray-on-gke-new-features-for-ai-scheduling-and-scaling-xwquzmlcf) · Google Cloud · 0 upvotes · 0 comments - [Run high-scale RL for LLMs on GKE](https://daily.dev/posts/run-high-scale-rl-for-llms-on-gke-oa4xx0efv) · Google Cloud · 0 upvotes · 0 comments - [In-House LLM Serving at Netflix](https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww) · Netflix TechBlog · 11 upvotes · 1 comments - [Running Ray at Scale on AKS](https://daily.dev/posts/running-ray-at-scale-on-aks-nhwwmuvcd) · InfoQ · 0 upvotes · 0 comments - [From notebooks to nodes: Architecting production-ready AI infrastructure](https://daily.dev/posts/from-notebooks-to-nodes-architecting-production-ready-ai-infrastructure-orotpqycd) · The New Stack · 1 upvotes · 0 comments --- Tags: [#gcp](https://daily.dev/tags/gcp), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/improving-ray-serve-llm-on-gke-throughput-latency-el4asvvua)