vLLM's engineering team details how they maintain production quality for the most widely used open-source LLM inference engine, handling 1,918 commits per month across 1,000+ model architectures and 600+ accelerator types. The post covers three layers: (1) CI with 266 jobs across 58 runner queues on a heterogeneous multi-provider hardware fleet, using consistent container images, pinned dependency lock files, MIG GPU slicing, autoscaling, and a custom CI dashboard at ci.vllm.ai backed by ClickHouse and Databricks; (2) nightly performance benchmarking and model accuracy evaluation across H200, B200, MI300X, and MI355X for models like DeepSeek V4 and gpt-oss, measuring TTFT, TPOT, and accuracy on GSM8K/GPQA/AIME/BFCL — a layer built after two emergency patches for v0.20.0 exposed gaps; and (3) a two-week release cadence with cherry-pick windows, multiple release candidates, and three mandatory gates before shipping 7 Python wheels and 11 Docker images. Future work includes automatic test selection, faster CI feedback under 30 minutes, and better flaky-test quarantine.
