The End of Standard Attention? | DeepSeek-V4 Explained

DeepSeek-V4 introduces a new architecture aimed at making million-token context windows practical for large language models. The key innovation is replacing standard full attention (which scales quadratically with sequence length) with two specialized mechanisms: Heavily Compressed Attention (HCA), which compresses groups of 128 tokens into a single entry, and Compressed Sparse Attention (CSA), which uses a gentler 4-to-1 compression followed by a Top-k selector to retain only the most relevant 1024 entries. Both mechanisms preserve recent local context by appending the last uncompressed tokens. The architecture also uses Manifold-Constrained Hyper-Connections (mHC) for a more expressive residual stream and DeepSeekMoE for feed-forward layers. At 1 million tokens, the KV cache is 9.5x smaller than DeepSeek-V3, with dramatic compute savings. A 'Max mode' further scales test-time compute via reinforcement learning. Benchmarks show DeepSeek-V4-Pro Max is competitive with top proprietary models and outperforms open-source alternatives, with strong multi-needle retrieval accuracy up to 128k tokens and reasonable performance up to 1 million tokens.