--- title: Netflix Details Its In-House LLM Serving Platform with... description: Netflix has shared production lessons from building its in-house LLM serving platform, combining its existing JVM-based serving layer with Triton and vLLM for... canonical: https://daily.dev/posts/netflix-details-its-in-house-llm-serving-platform-with-triton-and-vllm-8r6lrator twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Netflix Details Its In-House LLM Serving Platform with Triton and vLLM | daily.dev og:description: Netflix has shared production lessons from building its in-house LLM serving platform, combining its existing JVM-based serving layer with Triton and vLLM for... og:url: https://daily.dev/posts/netflix-details-its-in-house-llm-serving-platform-with-triton-and-vllm-8r6lrator og:image: https://api.daily.dev/og/posts/8R6lraTor.png og:image:alt: Netflix Details Its In-House LLM Serving Platform with Triton and vLLM og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Netflix Details Its In-House LLM Serving Platform with Triton and vLLM **[InfoQ](https://daily.dev/sources/infoq)** · 3 min read · 2 upvotes · 0 comments ## Summary Netflix has shared production lessons from building its in-house LLM serving platform, combining its existing JVM-based serving layer with Triton and vLLM for GPU inference. Smaller models run in-process on CPUs while larger ones are routed to a GPU path where Triton handles model management and scheduling and vLLM performs inference. Key challenges included version pinning between Triton and vLLM to prevent deployment failures, custom model integration via vLLM extension points when Hugging Face compatibility fell short, and maintaining state for constrained decoding when vLLM pauses and resumes requests. Netflix also compared Triton's Python backend versus its vLLM backend, finding the latter allows models and frontends to evolve more independently. Red-Black and Versioned deployment strategies manage model-level changes, keeping old and new revisions available during migrations. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why would Triton and vLLM be used together instead of just vLLM alone for serving large language models? Triton handles model loading, batching, GPU scheduling, and multi-framework serving management, while vLLM performs the actual inference and offers extension points for custom model architectures and decoding behavior. Netflix combines both because Triton controls the surrounding serving environment while vLLM's operational fit and extensibility make it better suited for the inference step itself. _daily.dev surfaces engineering writeups like this for teams weighing Triton plus vLLM against other serving stacks._ ### How does constrained decoding break when vLLM pauses and resumes a request to manage GPU resources? Constrained decoding filters which tokens a model may generate based on everything generated so far, requiring the decoder to maintain state throughout a request. When vLLM pauses and later resumes a request for GPU resource management, that state can fall out of sync with the token history. Netflix addressed this by adding logic to detect the mismatch and rebuild the decoder state before generation continues. _Developers debugging constrained decoding edge cases in production LLM systems can track approaches like this on daily.dev._ ### What deployment strategies handle incompatible schema changes when updating LLM models in production? Red-Black and Versioned deployment strategies manage changes at the model level. Versioned deployments keep old and new model revisions available separately, letting consuming applications migrate at their own pace after adapting to incompatible input or output schemas, rather than forcing an immediate cutover. _Teams planning zero-downtime model rollouts can find production patterns like these on daily.dev._ ## Similar posts on daily.dev - [In-House LLM Serving at Netflix](https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww) · Netflix TechBlog · 11 upvotes · 1 comments - [IBM Research uses vLLM at the heart of its RITS Platform – PyTorch](https://daily.dev/posts/ibm-research-uses-vllm-at-the-heart-of-its-rits-platform-pytorch-umwekupnu) · PyTorch · 0 upvotes · 0 comments - [Why vLLM is the best choice for AI inference today](https://daily.dev/posts/why-vllm-is-the-best-choice-for-ai-inference-today-7zxdzzpx7) · Red Hat Developer · 0 upvotes · 0 comments - [How Netflix Taught an LLM to Recommend Movies So That You Keep Watching](https://daily.dev/posts/how-netflix-taught-an-llm-to-recommend-movies-so-that-you-keep-watching-7nsw95tq8) · ByteByteGo · 58 upvotes · 8 comments --- Tags: [#netflix](https://daily.dev/tags/netflix), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/netflix-details-its-in-house-llm-serving-platform-with-triton-and-vllm-8r6lrator) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Netflix Details Its In-House LLM Serving Platform with Triton and vLLM","url":"https://daily.dev/posts/netflix-details-its-in-house-llm-serving-platform-with-triton-and-vllm-8r6lrator","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/netflix-details-its-in-house-llm-serving-platform-with-triton-and-vllm-8r6lrator"},"datePublished":"2026-07-27T07:46:08.653Z","dateModified":"2026-09-13T21:19:58.795Z","description":"Netflix has shared production lessons from building its in-house LLM serving platform, combining its existing JVM-based serving layer with Triton and vLLM for...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7a29369825186fd029568792792bdec0?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/7a29369825186fd029568792792bdec0?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"InfoQ","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"InfoQ","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/afc3bced3e1e4b188dd9127017a60e0c","url":"https://daily.dev/sources/infoq"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/netflix-details-its-in-house-llm-serving-platform-with-triton-and-vllm-8r6lrator","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"netflix,vllm","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"InfoQ","item":"https://daily.dev/sources/infoq"},{"@type":"ListItem","position":3,"name":"Netflix Details Its In-House LLM Serving Platform with Triton and vLLM"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/netflix-details-its-in-house-llm-serving-platform-with-triton-and-vllm-8r6lrator#faq","mainEntity":[{"@type":"Question","name":"Why would Triton and vLLM be used together instead of just vLLM alone for serving large language models?","acceptedAnswer":{"@type":"Answer","text":"Triton handles model loading, batching, GPU scheduling, and multi-framework serving management, while vLLM performs the actual inference and offers extension points for custom model architectures and decoding behavior. Netflix combines both because Triton controls the surrounding serving environment while vLLM's operational fit and extensibility make it better suited for the inference step itself. daily.dev surfaces engineering writeups like this for teams weighing Triton plus vLLM against other serving stacks."}},{"@type":"Question","name":"How does constrained decoding break when vLLM pauses and resumes a request to manage GPU resources?","acceptedAnswer":{"@type":"Answer","text":"Constrained decoding filters which tokens a model may generate based on everything generated so far, requiring the decoder to maintain state throughout a request. When vLLM pauses and later resumes a request for GPU resource management, that state can fall out of sync with the token history. Netflix addressed this by adding logic to detect the mismatch and rebuild the decoder state before generation continues. Developers debugging constrained decoding edge cases in production LLM systems can track approaches like this on daily.dev."}},{"@type":"Question","name":"What deployment strategies handle incompatible schema changes when updating LLM models in production?","acceptedAnswer":{"@type":"Answer","text":"Red-Black and Versioned deployment strategies manage changes at the model level. Versioned deployments keep old and new model revisions available separately, letting consuming applications migrate at their own pace after adapting to incompatible input or output schemas, rather than forcing an immediate cutover. Teams planning zero-downtime model rollouts can find production patterns like these on daily.dev."}}]} ```