A comprehensive guide to deploying the vLLM Production Stack on T Cloud Public CCE (Kubernetes). Covers four deployment patterns: single-GPU standalone serving, single-node tensor parallelism on NVIDIA T4 and V100 GPUs, multi-node distributed serving with Ray/KubeRay, and disaggregated prefill/decode serving using LMCache and NIXL for high-throughput workloads. Each scenario includes full Helm chart configurations, GPU memory considerations, quantization strategies (AWQ, GPTQ INT4), and validation steps. Models covered include GPT-OSS 20B, Qwen2.5-32B-Instruct-AWQ, Llama-3.1-70B-Instruct, and Qwen2.5-14B-Instruct-AWQ.
Table of contents
TL;DRIntroductionPrerequisitesCreating the SecretCreating the Persistence Volume ClaimInstalling KubeRay Operator (Optional)Choosing the Right Deployment ScenarioSingle-Node, Standalone ServingSingle-Node, Tensor Parallelism on NVIDIA T4/L4Single-Node, Tensor Parallelism on NVIDIA V100Get Akriotis Kyriakos ’s stories in your inboxMulti-Node, Distributed ServingDisaggregated Prefill/Decode ServingClosing Thoughts111 Impressions