In-House LLM Serving at Netflix

Netflix's AI Platform team shares how they built an in-house LLM serving platform on top of vLLM and NVIDIA Triton Inference Server, integrated into their existing production ML infrastructure. Key decisions covered include switching from TensorRT-LLM to vLLM as the primary engine, choosing between Triton's Python and vLLM backends for model packaging, exposing an OpenAI-compatible HTTP API alongside gRPC, and implementing Red-Black vs. versioned deployment strategies for zero-downtime rollouts. The post also details a deep-dive into constrained decoding at scale: an initial per-request Python logits processor hit GIL-induced CPU bottlenecks under batch load, which was resolved by migrating to vLLM V1's batch-level API and reimplementing the hot path in multi-threaded C++. Operational lessons include model caching on Amazon FSx to reduce cold-start latency, patching Triton's OpenAI frontend to properly pass response_format to vLLM's guided decoding, and building a unified Prometheus metrics proxy to merge vLLM and Triton observability.