--- title: Announcing New vLLM and SGLang Inference Engines on TPU description: Google launched native PyTorch backends for vLLM and SGLang on TPU, built on TorchTPU. vLLM-TorchTPU was co-developed with Inferact and SGLang-TorchTPU with... canonical: https://daily.dev/posts/announcing-new-vllm-and-sglang-inference-engines-on-tpu-s4veggelf twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Announcing New vLLM and SGLang Inference Engines on TPU | daily.dev og:description: Google launched native PyTorch backends for vLLM and SGLang on TPU, built on TorchTPU. vLLM-TorchTPU was co-developed with Inferact and SGLang-TorchTPU with... og:url: https://daily.dev/posts/announcing-new-vllm-and-sglang-inference-engines-on-tpu-s4veggelf og:image: https://api.daily.dev/og/posts/S4VeGGelf.png og:image:alt: Announcing New vLLM and SGLang Inference Engines on TPU og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Announcing New vLLM and SGLang Inference Engines on TPU **[Google Developers](https://daily.dev/sources/googledevs)** · 7 min read · 2 upvotes · 0 comments ## Summary Google launched native PyTorch backends for vLLM and SGLang on TPU, built on TorchTPU. vLLM-TorchTPU was co-developed with Inferact and SGLang-TorchTPU with RadixArk, the companies founded by each project's creators. Both engines use TPU Sync, an open-source C++ library, for KV cache transfer and management on TPU. On Ironwood TPU, the stack reached the lowest cost of 2.01 cents per million tokens at high-concurrency configurations, and a Pallas megakernel serves Kimi K3 at more than 700 tokens per second to a single user. A new community contribution TPU capacity pool offers free TPU compute to developers who actively contribute to both engines. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I run vLLM natively on Google Cloud TPUs with PyTorch instead of JAX? Google released vLLM-TorchTPU, a native PyTorch backend for vLLM built on TorchTPU, which makes the TPU a native PyTorch device. It was co-developed with Inferact, the company founded by vLLM's creators, and reuses vLLM's standard worker model, scheduler, KV cache manager and API server, adding TPU-specific kernels written in Pallas underneath. _Teams choosing a TPU inference stack can follow vLLM-TorchTPU developments on daily.dev._ ### What is TPU Sync and how does it handle KV cache transfer for disaggregated LLM serving? TPU Sync is an open-source C++ library for KV cache transfer and management on TPU, used by both vLLM-TorchTPU and SGLang-TorchTPU. It reads device buffers behind PyTorch tensors directly, avoiding host staging copies, and runs transfers on separate threads outside Python so disaggregated prefill and decode hosts can overlap transfer with computation. _Engineers building disaggregated inference pipelines can track KV cache tooling like this on daily.dev._ ### What throughput does the Kimi K3 model achieve on Google's Ironwood TPU with vLLM? A Pallas megakernel integrated into vLLM-TorchTPU runs the entire decode step for the 2.8-trillion-parameter Kimi K3 model as one kernel, serving more than 700 tokens per second to a single user on Ironwood TPU (TPU 7x). This was built with Inferact, and Ironwood also reached a cost of 2.01 cents per million tokens in high-concurrency benchmarks. _Anyone comparing TPU versus GPU inference economics can follow these benchmark results on daily.dev._ ## Similar posts on daily.dev - [Google splits its TPU line in two for the agentic era](https://daily.dev/posts/google-splits-its-tpu-line-in-two-for-the-agentic-era-zbjc2yyc7) · The New Stack · 0 upvotes · 0 comments - [Supercharging LLM inference on Google TPUs: Achieving 3X speedups with diffusion-style speculative decoding](https://daily.dev/posts/supercharging-llm-inference-on-google-tpus-achieving-3x-speedups-with-diffusion-style-speculative-d-jiflbzrv8) · Google Developers · 0 upvotes · 0 comments - [vLLM vs. SGLang: Choosing the right inference engine for your workload](https://daily.dev/posts/vllm-vs-sglang-choosing-the-right-inference-engine-for-your-workload-lirwkfyqx) · Red Hat Developer · 0 upvotes · 0 comments --- Tags: [#pytorch](https://daily.dev/tags/pytorch), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/announcing-new-vllm-and-sglang-inference-engines-on-tpu-s4veggelf) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Announcing New vLLM and SGLang Inference Engines on TPU","url":"https://daily.dev/posts/announcing-new-vllm-and-sglang-inference-engines-on-tpu-s4veggelf","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/announcing-new-vllm-and-sglang-inference-engines-on-tpu-s4veggelf"},"datePublished":"2026-10-20T15:08:36.131Z","dateModified":"2026-10-20T15:28:14.231Z","description":"Google launched native PyTorch backends for vLLM and SGLang on TPU, built on TorchTPU. vLLM-TorchTPU was co-developed with Inferact and SGLang-TorchTPU with...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/845936d77d7041649f5df5c2758ac5e6?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/845936d77d7041649f5df5c2758ac5e6?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Google Developers","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Google Developers","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/ea4bbded4e7f45ccb82e121a3156535f","url":"https://daily.dev/sources/googledevs"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/announcing-new-vllm-and-sglang-inference-engines-on-tpu-s4veggelf","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"pytorch,ai-inference,vllm","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Google Developers","item":"https://daily.dev/sources/googledevs"},{"@type":"ListItem","position":3,"name":"Announcing New vLLM and SGLang Inference Engines on TPU"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/announcing-new-vllm-and-sglang-inference-engines-on-tpu-s4veggelf#faq","mainEntity":[{"@type":"Question","name":"How do I run vLLM natively on Google Cloud TPUs with PyTorch instead of JAX?","acceptedAnswer":{"@type":"Answer","text":"Google released vLLM-TorchTPU, a native PyTorch backend for vLLM built on TorchTPU, which makes the TPU a native PyTorch device. It was co-developed with Inferact, the company founded by vLLM's creators, and reuses vLLM's standard worker model, scheduler, KV cache manager and API server, adding TPU-specific kernels written in Pallas underneath. Teams choosing a TPU inference stack can follow vLLM-TorchTPU developments on daily.dev."}},{"@type":"Question","name":"What is TPU Sync and how does it handle KV cache transfer for disaggregated LLM serving?","acceptedAnswer":{"@type":"Answer","text":"TPU Sync is an open-source C++ library for KV cache transfer and management on TPU, used by both vLLM-TorchTPU and SGLang-TorchTPU. It reads device buffers behind PyTorch tensors directly, avoiding host staging copies, and runs transfers on separate threads outside Python so disaggregated prefill and decode hosts can overlap transfer with computation. Engineers building disaggregated inference pipelines can track KV cache tooling like this on daily.dev."}},{"@type":"Question","name":"What throughput does the Kimi K3 model achieve on Google's Ironwood TPU with vLLM?","acceptedAnswer":{"@type":"Answer","text":"A Pallas megakernel integrated into vLLM-TorchTPU runs the entire decode step for the 2.8-trillion-parameter Kimi K3 model as one kernel, serving more than 700 tokens per second to a single user on Ironwood TPU (TPU 7x). This was built with Inferact, and Ironwood also reached a cost of 2.01 cents per million tokens in high-concurrency benchmarks. Anyone comparing TPU versus GPU inference economics can follow these benchmark results on daily.dev."}}]} ```