--- title: "Distributed Layerwise Offload: Scaling Toward 200B+ DiT Models Efficiently in vLLM-Omni" url: https://daily.dev/posts/distributed-layerwise-offload-scaling-toward-200b-dit-models-efficiently-in-vllm-omni-e6ycq2oue source_url: https://vllm.ai/blog/2026-08-17-distributed-layerwise-offload type: article source: "vLLM" published: 2026-08-17T08:47:15.767Z updated: 2026-08-17T08:47:50.620Z tags: ["gpu", "distributed-systems", "vllm", "diffusion-models"] reading_time: 20 upvotes: 1 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Distributed Layerwise Offload: Scaling Toward 200B+ DiT Models Efficiently in vLLM-Omni **[vLLM](https://daily.dev/sources/vllm)** · 20 min read · 1 upvotes · 0 comments ## Summary vLLM-Omni introduces Distributed Layerwise Offload (DLO), a technique that shards diffusion transformer (DiT) model weights across devices and streams them via AllGather, enabling models far larger than a single device's HBM (e.g., a 124 GB Cosmos3-Super model on 64 GB HBM cards) to run efficiently. Key techniques include meta-device initialization with mmap weight loading (cutting cold-start host memory peak by 73%), weight sharding with AllGather reconstruction, a fixed double-buffer scheme keeping only two layers on-device at a time, and DP multi-concurrency achieving 3.3x throughput over single-request HSDP. Benchmarks on Ascend 910B3 and NVIDIA B300 GPUs show correctness (byte-identical outputs) and significant memory savings versus HSDP, with a separate MiniMax-H3 study finding that the optimal DLO mode (AllGather vs. rank-local) depends on the DP/SP topology. The stack is presented as platform-agnostic across CUDA/NCCL and Ascend CANN/HCCL, with an extrapolation suggesting feasibility for 400 GB-class models on 2 TB RAM hosts. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What vLLM and vLLM-Omni versions do I need to use Distributed Layerwise Offload with AllGather in vLLM-Omni? The DLO plus AllGather quickstart requires vLLM 0.27.0 with vLLM-Omni v0.27.0rc1 or later. On the earlier v0.26.0 release, the Cosmos3 DLO plus DP path rejects every request because the engine requires supports_request_batch=True for multi-request admission, which the Cosmos3OmniDiffusersPipeline does not declare. PR #5864 fixes this by bypassing that requirement for DLO+AllGather+DP configurations. _Track version-specific gotchas like this one for vllm-omni upgrades on daily.dev before they break your pipeline._ ### How much does distributed layerwise offload reduce host memory versus traditional layerwise offload for a large diffusion model on multiple GPUs? For a 124 GB Cosmos3-Super model across 4 devices, traditional pure-DP layerwise offload stores a full model copy per rank, requiring 4 times 124 GB equals 496 GB of host RAM. Distributed Layerwise Offload shards weights so each rank stores only 1/dp_size, dropping total pinned memory to roughly 124 GB (about 31 GB per rank), while meta-device plus mmap loading also cuts cold-start cgroup peak by 73% (178 GB to 47 GB) on a smaller 33 GB model at DP4. _Compare memory tradeoffs across GPU serving strategies like this on daily.dev when scaling large models._ ### When should I use AllGather-based DLO versus rank-local DLO for multi-GPU diffusion model serving? AllGather DLO wins at lower DP degrees: on an 8x B300 node, it improves throughput by 129.4% and cuts P50 latency by 56.6% at DP1xSP8, but the benefit narrows to just 2.2% at DP4xSP2. At DP8xSP1, AllGather actually reduces throughput by 4.1%, raises latency by 3.8%, and increases per-GPU peak memory from 20.03 to 94.03 GiB, so rank-local DLO is preferred there instead. _Follow topology-dependent benchmarks like this on daily.dev before picking a multi-GPU serving strategy._ ## Similar posts on daily.dev - [vLLM-Omni Diffusion Cache Acceleration](https://daily.dev/posts/vllm-omni-diffusion-cache-acceleration-4s7vln3xq) · vLLM · 1 upvotes · 0 comments - [Driving vLLM WideEP and Large-Scale Serving Toward Maturity on Blackwell \(Part I\)](https://daily.dev/posts/driving-vllm-wideep-and-large-scale-serving-toward-maturity-on-blackwell-part-i--36r4rera9) · vLLM · 1 upvotes · 0 comments - [Disaggregated Serving for Hybrid SSM Models in vLLM](https://daily.dev/posts/disaggregated-serving-for-hybrid-ssm-models-in-vllm-vqjdefulm) · vLLM · 0 upvotes · 0 comments --- Tags: [#gpu](https://daily.dev/tags/gpu), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#vllm](https://daily.dev/tags/vllm), [#diffusion-models](https://daily.dev/tags/diffusion-models) [View this post on daily.dev](https://daily.dev/posts/distributed-layerwise-offload-scaling-toward-200b-dit-models-efficiently-in-vllm-omni-e6ycq2oue)