The Insane Infrastructure Design of DeepSeek V4

A deep technical breakdown of DeepSeek V4's infrastructure design, covering the full stack of optimizations that make million-token context cheap and efficient. Topics include the complex multi-branch attention path (CSA, HCA, compressed KV, partial RoPE, FP4 quantization), the MegaE kernel that fuses expert dispatch/compute/combine into a single GPU program for 1.5–2x inference speedup, a dual-kernel strategy for batch-invariant attention decoding, on-disk KV cache storage enabling near-100% cache hit rates, a hybrid Zero+Muon optimizer sharding strategy for 1.6T parameter training, teacher scheduling for multi-specialist distillation, a token-level write-ahead log for resumable RL rollouts, and the DeepSeek Elastic Compute (DEC) sandbox platform supporting function calls, containers, microVMs, and full VMs for agentic training at scale.