vLLM-Omni introduces Distributed Layerwise Offload (DLO), a technique that shards diffusion transformer (DiT) model weights across devices and streams them via AllGather, enabling models far larger than a single device's HBM (e.g., a 124 GB Cosmos3-Super model on 64 GB HBM cards) to run efficiently. Key techniques include meta-device initialization with mmap weight loading (cutting cold-start host memory peak by 73%), weight sharding with AllGather reconstruction, a fixed double-buffer scheme keeping only two layers on-device at a time, and DP multi-concurrency achieving 3.3x throughput over single-request HSDP. Benchmarks on Ascend 910B3 and NVIDIA B300 GPUs show correctness (byte-identical outputs) and significant memory savings versus HSDP, with a separate MiniMax-H3 study finding that the optimal DLO mode (AllGather vs. rank-local) depends on the DP/SP topology. The stack is presented as platform-agnostic across CUDA/NCCL and Ascend CANN/HCCL, with an extrapolation suggesting feasibility for 400 GB-class models on 2 TB RAM hosts.
Table of contents
TL;DRQuickstartThe Problem: Large Diffusion Models vs. HBM and Host MemorySolution Overview1. Meta Device + mmap Weight Loading2. Weight Sharding with AllGather Reconstruction3. Double-Buffered Prefetch with H2D + AllGather Overlap4. DP Multi-Concurrency: N Requests in ParallelAscend Memory Accounting: cgroup-visible vs. Physical RAMValidation ResultsAcknowledgementsReferencesQuestions this post answers
What vLLM and vLLM-Omni versions do I need to use Distributed Layerwise Offload with AllGather in vLLM-Omni?
The DLO plus AllGather quickstart requires vLLM 0.27.0 with vLLM-Omni v0.27.0rc1 or later. On the earlier v0.26.0 release, the Cosmos3 DLO plus DP path rejects every request because the engine requires supports_request_batch=True for multi-request admission, which the Cosmos3OmniDiffusersPipeline does not declare. PR #5864 fixes this by bypassing that requirement for DLO+AllGather+DP configurations. Track version-specific gotchas like this one for vllm-omni upgrades on daily.dev before they break your pipeline.
How much does distributed layerwise offload reduce host memory versus traditional layerwise offload for a large diffusion model on multiple GPUs?
For a 124 GB Cosmos3-Super model across 4 devices, traditional pure-DP layerwise offload stores a full model copy per rank, requiring 4 times 124 GB equals 496 GB of host RAM. Distributed Layerwise Offload shards weights so each rank stores only 1/dp_size, dropping total pinned memory to roughly 124 GB (about 31 GB per rank), while meta-device plus mmap loading also cuts cold-start cgroup peak by 73% (178 GB to 47 GB) on a smaller 33 GB model at DP4. Compare memory tradeoffs across GPU serving strategies like this on daily.dev when scaling large models.
When should I use AllGather-based DLO versus rank-local DLO for multi-GPU diffusion model serving?
AllGather DLO wins at lower DP degrees: on an 8x B300 node, it improves throughput by 129.4% and cuts P50 latency by 56.6% at DP1xSP8, but the benefit narrows to just 2.2% at DP4xSP2. At DP8xSP1, AllGather actually reduces throughput by 4.1%, raises latency by 3.8%, and increases per-GPU peak memory from 20.03 to 94.03 GiB, so rank-local DLO is preferred there instead. Follow topology-dependent benchmarks like this on daily.dev before picking a multi-GPU serving strategy.