<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/llm-d-1-0-an-open-source-stack-for-distributed-inference-5yvf9dd61" -->

---
title: llm-d 1.0, an open source stack for distributed inference
description: Google announced llm-d 1.0, an open-source, Kubernetes-native stack for distributed LLM inference co-founded with Red Hat, IBM and over 500 contributors across...
canonical: https://daily.dev/posts/llm-d-1-0-an-open-source-stack-for-distributed-inference-5yvf9dd61
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: llm-d 1.0, an open source stack for distributed inference | daily.dev
og:description: Google announced llm-d 1.0, an open-source, Kubernetes-native stack for distributed LLM inference co-founded with Red Hat, IBM and over 500 contributors across...
og:url: https://daily.dev/posts/llm-d-1-0-an-open-source-stack-for-distributed-inference-5yvf9dd61
og:image: https://api.daily.dev/og/posts/5yVF9DD61.png
og:image:alt: llm-d 1.0, an open source stack for distributed inference
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# llm-d 1.0, an open source stack for distributed inference

**[Google Cloud](https://daily.dev/sources/gcp)** · 10 min read · 3 upvotes · 1 comments

## Summary

Google announced llm-d 1.0, an open-source, Kubernetes-native stack for distributed LLM inference co-founded with Red Hat, IBM and over 500 contributors across 25 organizations, released at the PyTorch Conference as a CNCF Sandbox project. The stack provides cache-aware routing, tiered KV cache offloading, prefill/decode disaggregation, multi-tenant flow control, and zero-downtime upgrades via DisaggregatedSet, designed to serve multi-step agent and RL workloads across mixed Google Cloud TPU and GPU fleets. Google reports production results including Gemini doubling prefix cache hit rate from 35% to 70% and cutting TTFT over 35%, a 17,000-node multi-region GKE deployment scaling throughput 2.9x, and customers like Snap, Mistral, and Zenteiq reporting cache hit rates up to 80% and 4x throughput gains.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://cloud.google.com/blog/topics/ai-infrastructure/llm-d-1-0-an-open-source-stack-for-distributed-inference>

## Questions this post answers

### What is llm-d and how does it help serve LLM inference at scale on Kubernetes?

llm-d is an open-source, Kubernetes-native stack for distributed LLM inference, co-founded by Google with Red Hat, IBM, and over 500 contributors from 25 organizations as a CNCF Sandbox project. It turns a cluster of model servers into a coordinated pool with cache-aware routing, tiered KV cache offloading, prefill/decode disaggregation, and multi-tenant flow control, built on vLLM, SGLang, and Kubernetes primitives like Gateway API and LeaderWorkerSet.

_Teams choosing between build-your-own and off-the-shelf inference serving can track llm-d developments on daily.dev._

### How much does prefix-aware routing and KV cache offloading improve LLM inference latency in production?

On an agentic coding workload on Ironwood TPU, prefix-aware routing combined with host memory offload cut median time-to-first-token by more than 85% and raised total throughput 2.4x compared to a plain Kubernetes Service. Separately, Google's Gemini platform doubled its prefix cache hit rate from 35% to 70% using GKE Inference Gateway, cutting TTFT over 35% for coding agents and P95 TTFT over 50% under bursty chat traffic.

_Engineers tuning inference latency can follow real-world KV cache offloading benchmarks on daily.dev._

### How does llm-d's DisaggregatedSet enable zero-downtime upgrades for prefill/decode disaggregated inference?

DisaggregatedSet, built on the Kubernetes LeaderWorkerSet API, orchestrates prefill and decode pools as one unit so rolling updates move both phases together on matching versions while each phase scales independently, without dropping context or disrupting active sessions. This avoids the problem of running prefill (compute-heavy) and decode (memory-bound) on the same replicas, which forces each to wait on the other.

_Platform teams planning safe rollouts of inference infrastructure can follow disaggregated serving patterns on daily.dev._

## Community discussion

Top comments from developers on daily.dev.

**@petecapecod** · 1 upvotes

> woah this is **awesome!!** Seems pretty big as far as teams or companies being able to set up and route reqs to the appropriate models without cache or KV loss.
>
> Very cool

## Similar posts on daily.dev

- [Introduction to distributed inference with llm-d](https://daily.dev/posts/introduction-to-distributed-inference-with-llm-d-ed6jsysii) · Red Hat Developer · 1 upvotes · 0 comments
- [Understanding disaggregated GenAI model serving with llm-d](https://daily.dev/posts/understanding-disaggregated-genai-model-serving-with-llm-d-lvuvnpaha) · Ubuntu · 1 upvotes · 0 comments

---

Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#gcp](https://daily.dev/tags/gcp), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm)

[View this post on daily.dev](https://daily.dev/posts/llm-d-1-0-an-open-source-stack-for-distributed-inference-5yvf9dd61)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"llm-d 1.0, an open source stack for distributed inference","url":"https://daily.dev/posts/llm-d-1-0-an-open-source-stack-for-distributed-inference-5yvf9dd61","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/llm-d-1-0-an-open-source-stack-for-distributed-inference-5yvf9dd61"},"datePublished":"2026-10-20T11:40:46.444Z","dateModified":"2026-10-20T11:41:49.360Z","description":"Google announced llm-d 1.0, an open-source, Kubernetes-native stack for distributed LLM inference co-founded with Red Hat, IBM and over 500 contributors across...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fae798aa3baebc72680bb6fa8c180d93?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fae798aa3baebc72680bb6fa8c180d93?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Google Cloud","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Google Cloud","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/gcp","url":"https://daily.dev/sources/gcp"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/llm-d-1-0-an-open-source-stack-for-distributed-inference-5yvf9dd61","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"kubernetes,gcp,ai-inference,vllm","timeRequired":"PT10M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Google Cloud","item":"https://daily.dev/sources/gcp"},{"@type":"ListItem","position":3,"name":"llm-d 1.0, an open source stack for distributed inference"}]}
{"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/llm-d-1-0-an-open-source-stack-for-distributed-inference-5yvf9dd61","comment":[{"@type":"Comment","text":"woah this is awesome!! Seems pretty big as far as teams or companies being able to set up and route reqs to the appropriate models without cache or KV loss.\nVery cool","datePublished":"2026-10-20T15:41:14.616Z","url":"https://daily.dev/posts/5yVF9DD61#c-Itp41LZP5","author":{"@type":"Person","name":"Peter Cruckshank","url":"https://daily.dev/petecapecod","image":"https://media.daily.dev/image/upload/s--ZJhQyKws--/f_auto/v1721235024/avatars/avatar_A9xh33q0QoxtkGoJRCosp"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/llm-d-1-0-an-open-source-stack-for-distributed-inference-5yvf9dd61#faq","mainEntity":[{"@type":"Question","name":"What is llm-d and how does it help serve LLM inference at scale on Kubernetes?","acceptedAnswer":{"@type":"Answer","text":"llm-d is an open-source, Kubernetes-native stack for distributed LLM inference, co-founded by Google with Red Hat, IBM, and over 500 contributors from 25 organizations as a CNCF Sandbox project. It turns a cluster of model servers into a coordinated pool with cache-aware routing, tiered KV cache offloading, prefill/decode disaggregation, and multi-tenant flow control, built on vLLM, SGLang, and Kubernetes primitives like Gateway API and LeaderWorkerSet. Teams choosing between build-your-own and off-the-shelf inference serving can track llm-d developments on daily.dev."}},{"@type":"Question","name":"How much does prefix-aware routing and KV cache offloading improve LLM inference latency in production?","acceptedAnswer":{"@type":"Answer","text":"On an agentic coding workload on Ironwood TPU, prefix-aware routing combined with host memory offload cut median time-to-first-token by more than 85% and raised total throughput 2.4x compared to a plain Kubernetes Service. Separately, Google's Gemini platform doubled its prefix cache hit rate from 35% to 70% using GKE Inference Gateway, cutting TTFT over 35% for coding agents and P95 TTFT over 50% under bursty chat traffic. Engineers tuning inference latency can follow real-world KV cache offloading benchmarks on daily.dev."}},{"@type":"Question","name":"How does llm-d's DisaggregatedSet enable zero-downtime upgrades for prefill/decode disaggregated inference?","acceptedAnswer":{"@type":"Answer","text":"DisaggregatedSet, built on the Kubernetes LeaderWorkerSet API, orchestrates prefill and decode pools as one unit so rolling updates move both phases together on matching versions while each phase scales independently, without dropping context or disrupting active sessions. This avoids the problem of running prefill (compute-heavy) and decode (memory-bound) on the same replicas, which forces each to wait on the other. Platform teams planning safe rollouts of inference infrastructure can follow disaggregated serving patterns on daily.dev."}}]}
```