vLLM-Omni serves Qwen3-Omni as a three-stage pipeline (Thinker → Talker → Code2Wav) and applies a layered set of optimizations to improve throughput and latency for online speech generation workloads. The post walks through each optimization in order: stage decomposition with per-stage batching as the baseline, CUDA Graph capture per stage (yielding ~4× throughput jump), async chunk handoffs to pipeline inter-stage transfers (largest audio TTFP reduction, from 2790ms to 655ms), async output for non-blocking payload construction, stage replicas to scale only the bottleneck Talker/Code2Wav stages, and hot-path cleanup targeting per-step Python/allocation overhead. Combined, these bring throughput from 2.2 to 11.7 req/s at concurrency 64, audio TTFP from ~5884ms to ~632ms, and audio RTF from 1.15 to 0.47 — moving from above-real-time to comfortably below it.
