DeepSeek-V4 introduces a hybrid attention design (CSA, HCA, SWA) that compresses context along the token axis before KV storage, making 1M-token context windows more practical as a serving workload. The post details the inference engineering challenges this creates: managing three distinct KV cache layouts per request, rethinking prefix caching as a per-cache-type storage policy, and tuning endpoint profiles for different workload shapes. On NVIDIA HGX B200, Together AI found that smart cache eviction policies (not just the architecture itself) drove the biggest gains — increasing total KV-cache capacity from ~1.2M to ~3.7M tokens on a single node. Performance is regime-dependent: long-context decode-heavy workloads benefit most immediately, while short-context prefill-heavy workloads are more sensitive to kernel maturity. Practical benchmarking guidance covers context-length regime, prefix reuse, cache policy, and endpoint profile selection.
