vLLM-Omni extends the vLLM serving engine to handle multimodal output models like Qwen3-Omni, which produce text, audio, and images rather than just text tokens. The architecture decomposes inference into a graph of stages — Thinker (~30B MoE), Talker (~3B MoE), and Code2Wav vocoder — each with its own GPU memory budget and independent scaling. Key features include a single OpenAI-compatible endpoint, inherited vLLM primitives (PagedAttention, continuous batching, prefix caching extended to hidden-state tensors), shared-memory transport via OmniConnector, and async chunked pipeline execution that lets stages overlap so audio streams out before earlier stages finish. A demo on a single NVIDIA B200 shows an insurance claim triage use case with concurrent adjuster and customer-callback requests. Benchmarks against Hugging Face Transformers show vLLM-Omni achieves a real-time factor below 1.0 for audio generation versus 2.64 for the baseline. The engine also supports diffusion model stages alongside autoregressive ones on the same abstractions.
