--- title: "In-House LLM Serving at Netflix" url: https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww source_url: https://netflixtechblog.com/in-house-llm-serving-at-netflix-a5a8e799ea2c type: article source: "Netflix TechBlog" published: 2026-07-18T07:23:36.368Z updated: 2026-07-18T10:20:59.859Z tags: ["mlops", "vllm"] reading_time: 12 upvotes: 11 comments: 1 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # In-House LLM Serving at Netflix **[Netflix TechBlog](https://daily.dev/sources/netflix)** · 12 min read · 11 upvotes · 1 comments ## Summary Netflix's AI Platform team shares how they built an in-house LLM serving platform on top of vLLM and NVIDIA Triton Inference Server, integrated into their existing production ML infrastructure. Key decisions covered include switching from TensorRT-LLM to vLLM as the primary engine, choosing between Triton's Python and vLLM backends for model packaging, exposing an OpenAI-compatible HTTP API alongside gRPC, and implementing Red-Black vs. versioned deployment strategies for zero-downtime rollouts. The post also details a deep-dive into constrained decoding at scale: an initial per-request Python logits processor hit GIL-induced CPU bottlenecks under batch load, which was resolved by migrating to vLLM V1's batch-level API and reimplementing the hot path in multi-threaded C++. Operational lessons include model caching on Amazon FSx to reduce cold-start latency, patching Triton's OpenAI frontend to properly pass response_format to vLLM's guided decoding, and building a unified Prometheus metrics proxy to merge vLLM and Triton observability. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community discussion Top comments from developers on daily.dev. **@kartiknvj** · 0 upvotes > The constrained-decoding detail is the part I found most useful, since a per-request Python logits processor hitting a GIL bottleneck under batch load is a trap I would not have predicted before profiling. Moving the hot path to the batch-level API and reimplementing in C++ makes sense, but I am curious how much guided decoding cost you in tokens-per-second once the schema got complex. Merging the two engines behind one Prometheus proxy also seems underrated, because inconsistent metric labels across serving backends quietly break every latency dashboard. ## Similar posts on daily.dev - [Netflix Details Its In-House LLM Serving Platform with Triton and vLLM](https://daily.dev/posts/netflix-details-its-in-house-llm-serving-platform-with-triton-and-vllm-8r6lrator) · InfoQ · 2 upvotes · 0 comments --- Tags: [#mlops](https://daily.dev/tags/mlops), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww)