--- title: "Efficient Decode Context Parallelism with vLLM for Long Context Workloads" url: https://daily.dev/posts/efficient-decode-context-parallelism-with-vllm-for-long-context-workloads-rae4d5gxo source_url: https://vllm.ai/blog/2026-08-07-decode-context-parallelism type: article source: "vLLM" published: 2026-08-07T19:43:46.365Z updated: 2026-08-07T19:44:21.827Z tags: ["ai-inference", "vllm"] reading_time: 11 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Efficient Decode Context Parallelism with vLLM for Long Context Workloads **[vLLM](https://daily.dev/sources/vllm)** · 11 min read · 0 upvotes · 0 comments ## Summary Decode Context Parallelism (DCP) in vLLM addresses a fundamental bottleneck in long-context LLM inference: KV cache replication across GPUs under tensor parallelism. Instead of partitioning by attention head (which hits a floor with GQA and is useless for MLA), DCP shards the KV cache along the sequence dimension so each GPU stores only 1/N of every request's KV data. Benchmarks on an 8×B200 node serving Kimi K2.6 show DCP reaching 6,091 tok/s/GPU at concurrency 512 with 82% KV usage, versus a baseline TP ceiling of ~1,863 tok/s/GPU that maxes out memory at concurrency 64. The post explains the AllGather Q → Compute → AllGather+ReduceScatter communication pattern, covers MLA and GQA backend constraints, and provides usage via the `--decode-context-parallel-size` flag. Future work includes better A2A kernels, speculative decoding support, and prefill/decode disaggregation hardening. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does Decode Context Parallelism in vLLM differ from tensor parallelism for long-context inference? Tensor parallelism partitions the KV cache by attention head, hitting a hard floor once GPU count exceeds the number of KV heads — causing full cache replication. DCP instead shards the KV cache along the sequence dimension, so each GPU stores only 1/N of every request's tokens. On an 8×B200 node with Kimi K2.6, this lets DCP reach 6,091 tok/s/GPU at concurrency 512 while TP plateaus at ~1,863 tok/s/GPU and runs out of memory at concurrency 64. _Engineers scaling long-context serving beyond a single TP group track DCP developments on daily.dev._ ### What are the constraints for enabling decode_context_parallel_size in vLLM with MLA models like DeepSeek or Kimi? For MLA models (DeepSeek-V2/V3/R1, Kimi K2.6), MLA compresses all KV into a single latent vector — effectively one KV head — so TP cannot shrink it. DCP sequence-splits that latent instead. The constraints are: tensor_parallel_size >= decode_context_parallel_size, and tensor_parallel_size % decode_context_parallel_size == 0. An optional flag VLLM_DCP_Q_REPLICATE=1 can skip the query all-gather at decode time for MLA. _Developers deploying MLA-based models in production find constraint details like these on daily.dev before they hit runtime errors._ ### What is the communication pattern used by Decode Context Parallelism during the decode phase in vLLM? DCP follows an AllGather Q → Compute → AllGather + ReduceScatter rhythm. First, each GPU all-gathers the full query vector (cheap at decode since it is a single token). Then each GPU runs attention against its local KV cache slice. Finally, partial attention outputs and log-sum-exp values are combined via AllGather and merged using the online-softmax trick, with ReduceScatter returning each GPU its own head-slice of the final output. _Teams optimizing LLM serving latency stay on top of inference kernel changes like these through daily.dev._ ## Similar posts on daily.dev - [Speeding Up Variable-Length Training with Dynamic Context Parallelism and NVIDIA Megatron Core](https://daily.dev/posts/speeding-up-variable-length-training-with-dynamic-context-parallelism-and-nvidia-megatron-core-wpjwnkyfv) · NVIDIA Developer · 1 upvotes · 0 comments - [Designing distributed AI inference: Core concepts and scaling dimensions](https://daily.dev/posts/designing-distributed-ai-inference-core-concepts-and-scaling-dimensions-7bcq8ydb3) · Red Hat Developer · 0 upvotes · 0 comments --- Tags: [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/efficient-decode-context-parallelism-with-vllm-for-long-context-workloads-rae4d5gxo)