--- title: In-House LLM Serving at Netflix | daily.dev description: Netflix's AI Platform team shares how they built an in-house LLM serving platform on top of vLLM and NVIDIA Triton Inference Server, integrated into their... canonical: https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: In-House LLM Serving at Netflix | daily.dev og:description: Netflix's AI Platform team shares how they built an in-house LLM serving platform on top of vLLM and NVIDIA Triton Inference Server, integrated into their... og:url: https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww og:image: https://api.daily.dev/og/posts/yh81G6jww.png og:image:alt: In-House LLM Serving at Netflix og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # In-House LLM Serving at Netflix **[Netflix TechBlog](https://daily.dev/sources/netflix)** · 12 min read · 11 upvotes · 1 comments ## Summary Netflix's AI Platform team shares how they built an in-house LLM serving platform on top of vLLM and NVIDIA Triton Inference Server, integrated into their existing production ML infrastructure. Key decisions covered include switching from TensorRT-LLM to vLLM as the primary engine, choosing between Triton's Python and vLLM backends for model packaging, exposing an OpenAI-compatible HTTP API alongside gRPC, and implementing Red-Black vs. versioned deployment strategies for zero-downtime rollouts. The post also details a deep-dive into constrained decoding at scale: an initial per-request Python logits processor hit GIL-induced CPU bottlenecks under batch load, which was resolved by migrating to vLLM V1's batch-level API and reimplementing the hot path in multi-threaded C++. Operational lessons include model caching on Amazon FSx to reduce cold-start latency, patching Triton's OpenAI frontend to properly pass response_format to vLLM's guided decoding, and building a unified Prometheus metrics proxy to merge vLLM and Triton observability. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does Triton's vLLM backend fail to load with a vllm.engine.metrics import error? This happens due to a version mismatch between Triton and vLLM: Triton's vLLM backend is compiled against a specific vLLM API surface, and when versions drift, imports break. For example, Triton 25.09 imports vllm.engine.metrics, a module that was removed in vLLM 0.11.2, causing the backend to fail to load entirely. The fix is pinning compatible Triton and vLLM versions when building the service image and preventing model authors from overriding the vLLM version at packaging time. _Teams pinning Triton and vLLM versions together can track compatibility gotchas like this one on daily.dev._ ### Why did custom logits processors in vLLM V0 not scale with batch size? In vLLM V0, custom logits processors run per-request rather than at the batch level: the CPU waits for GPU logits to transfer, then constraint logic runs sequentially per request because Python's GIL prevents parallelizing that work. CPU time in logit processing grows linearly with batch size, making end-to-end latency CPU-bound even though the model's forward pass is batched efficiently on GPU. vLLM V1 fixed this by moving logits processing to the batch level, allowing a rewritten processor in multi-threaded C++ to keep processing time flat as batch size grows. _Anyone scaling custom vLLM decoding logic can follow engine-version tradeoffs like V0 versus V1 on daily.dev._ ### How can I expose an OpenAI-compatible API on top of NVIDIA Triton Inference Server for a custom LLM serving stack? NVIDIA provides a Triton OpenAI-compatible frontend that starts an embedded Triton server, wraps it in an engine class converting request schemas into Triton inference requests, and serves responses through FastAPI, with KServe HTTP/gRPC frontends enabled alongside for existing clients. One gap to patch manually: the response_format field in the schema was silently dropped before reaching vLLM, so requests for JSON output could return malformed JSON with guided decoding never applied unless the frontend is patched to translate response_format into vLLM's guided decoding parameters. _Developers wiring OpenAI-compatible APIs onto self-hosted inference engines can compare setups like this on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@kartiknvj** · 0 upvotes > The constrained-decoding detail is the part I found most useful, since a per-request Python logits processor hitting a GIL bottleneck under batch load is a trap I would not have predicted before profiling. Moving the hot path to the batch-level API and reimplementing in C++ makes sense, but I am curious how much guided decoding cost you in tokens-per-second once the schema got complex. Merging the two engines behind one Prometheus proxy also seems underrated, because inconsistent metric labels across serving backends quietly break every latency dashboard. ## Similar posts on daily.dev - [Netflix Details Its In-House LLM Serving Platform with Triton and vLLM](https://daily.dev/posts/netflix-details-its-in-house-llm-serving-platform-with-triton-and-vllm-8r6lrator) · InfoQ · 2 upvotes · 0 comments --- Tags: [#mlops](https://daily.dev/tags/mlops), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"In-House LLM Serving at Netflix","url":"https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww"},"datePublished":"2026-07-18T07:23:36.368Z","dateModified":"2026-09-14T06:38:45.591Z","description":"Netflix's AI Platform team shares how they built an in-house LLM serving platform on top of vLLM and NVIDIA Triton Inference Server, integrated into their...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/167c52d10b2666f4fa8f6a7a02eb4a76?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/167c52d10b2666f4fa8f6a7a02eb4a76?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Netflix TechBlog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Netflix TechBlog","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/netflix","url":"https://daily.dev/sources/netflix"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":11},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"mlops,vllm","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Netflix TechBlog","item":"https://daily.dev/sources/netflix"},{"@type":"ListItem","position":3,"name":"In-House LLM Serving at Netflix"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww","comment":[{"@type":"Comment","text":"The constrained-decoding detail is the part I found most useful, since a per-request Python logits processor hitting a GIL bottleneck under batch load is a trap I would not have predicted before profiling. Moving the hot path to the batch-level API and reimplementing in C++ makes sense, but I am curious how much guided decoding cost you in tokens-per-second once the schema got complex. Merging the two engines behind one Prometheus proxy also seems underrated, because inconsistent metric labels across serving backends quietly break every latency dashboard.","datePublished":"2026-07-20T16:21:27.374Z","url":"https://daily.dev/posts/yh81G6jww#c-Q6k09L4Ox","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/in-house-llm-serving-at-netflix-yh81g6jww#faq","mainEntity":[{"@type":"Question","name":"Why does Triton's vLLM backend fail to load with a vllm.engine.metrics import error?","acceptedAnswer":{"@type":"Answer","text":"This happens due to a version mismatch between Triton and vLLM: Triton's vLLM backend is compiled against a specific vLLM API surface, and when versions drift, imports break. For example, Triton 25.09 imports vllm.engine.metrics, a module that was removed in vLLM 0.11.2, causing the backend to fail to load entirely. The fix is pinning compatible Triton and vLLM versions when building the service image and preventing model authors from overriding the vLLM version at packaging time. Teams pinning Triton and vLLM versions together can track compatibility gotchas like this one on daily.dev."}},{"@type":"Question","name":"Why did custom logits processors in vLLM V0 not scale with batch size?","acceptedAnswer":{"@type":"Answer","text":"In vLLM V0, custom logits processors run per-request rather than at the batch level: the CPU waits for GPU logits to transfer, then constraint logic runs sequentially per request because Python's GIL prevents parallelizing that work. CPU time in logit processing grows linearly with batch size, making end-to-end latency CPU-bound even though the model's forward pass is batched efficiently on GPU. vLLM V1 fixed this by moving logits processing to the batch level, allowing a rewritten processor in multi-threaded C++ to keep processing time flat as batch size grows. Anyone scaling custom vLLM decoding logic can follow engine-version tradeoffs like V0 versus V1 on daily.dev."}},{"@type":"Question","name":"How can I expose an OpenAI-compatible API on top of NVIDIA Triton Inference Server for a custom LLM serving stack?","acceptedAnswer":{"@type":"Answer","text":"NVIDIA provides a Triton OpenAI-compatible frontend that starts an embedded Triton server, wraps it in an engine class converting request schemas into Triton inference requests, and serves responses through FastAPI, with KServe HTTP/gRPC frontends enabled alongside for existing clients. One gap to patch manually: the response_format field in the schema was silently dropped before reaching vLLM, so requests for JSON output could return malformed JSON with guided decoding never applied unless the frontend is patched to translate response_format into vLLM's guided decoding parameters. Developers wiring OpenAI-compatible APIs onto self-hosted inference engines can compare setups like this on daily.dev."}}]} ```