--- title: DeepSeek-V4 preview: two MoE models with 1M context and... description: DeepSeek released two new open-weight models under the V4 banner: V4-Pro (1.6T parameters, 49B activated) and V4-Flash (284B, 13B activated), both supporting... canonical: https://daily.dev/posts/deepseek-v4-preview-two-moe-models-with-1m-context-and-new-attention-mechanism-pbpwljmj3 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: DeepSeek-V4 preview: two MoE models with 1M context and new attention mechanism | daily.dev og:description: DeepSeek released two new open-weight models under the V4 banner: V4-Pro (1.6T parameters, 49B activated) and V4-Flash (284B, 13B activated), both supporting... og:url: https://daily.dev/posts/deepseek-v4-preview-two-moe-models-with-1m-context-and-new-attention-mechanism-pbpwljmj3 og:image: https://api.daily.dev/og/posts/PBpWLJmJ3.png og:image:alt: DeepSeek-V4 preview: two MoE models with 1M context and new attention mechanism og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # DeepSeek-V4 preview: two MoE models with 1M context and new attention mechanism **[Collections](https://daily.dev/sources/collections)** · 4 min read · 2 upvotes · 0 comments ## Summary DeepSeek released two new open-weight models under the V4 banner: V4-Pro (1.6T parameters, 49B activated) and V4-Flash (284B, 13B activated), both supporting 1M token context windows under MIT License. The key architectural innovation is a hybrid attention system combining token-level compression, Heavily Compressed Attention (128:1 ratio), and Compressed Sparse Attention, reducing KV cache memory by ~90% vs V3.2. V4-Pro scores 93.5 on LiveCodeBench and costs ~14% of Claude Opus 4.7, while Flash costs just $0.02 per complex task attempt. Training uses the Muon optimizer borrowed from Kimi K2. vLLM now supports both models with kernel fusions delivering 1.4–20x speedups. A local inference engine called DS4 (built on llama.cpp/GGML) runs Flash on an M3 Max 128GB at mixed 2/4-bit quantization, enabling features like activation steering and disk-based KV cache sessions. ## Content ## What DeepSeek V4 actually is DeepSeek released two models under the V4 banner: **DeepSeek-V4-Pro** (1.6T total parameters, 49B activated) and **DeepSeek-V4-Flash** (284B total, 13B activated). Both support 1 million token context windows and are released under the MIT License. The Pro model matches several frontier closed-source models on coding benchmarks — 93.5 on LiveCodeBench, 3206 Codeforces rating — while Flash runs at roughly 8% of Pro's cost, which itself is about 14% of Claude Opus 4.7's cost. Neither model supports images or audio. There's some degradation near the context window limits, and the paper is candid that some training stabilization techniques aren't fully understood even by the authors. ## The attention architecture The headline technical contribution is a hybrid attention system that combines three compression layers: - **Token-level compression** via shared key-value vectors - **Heavily Compressed Attention (HCA)** — 128-to-1 compression ratio - **Compressed Sparse Attention (CSA)** — 4x compression with sparse attention patterns and a short sliding window for locality Together these reduce KV cache memory by roughly 90% compared to DeepSeek-V3.2 at 1M context, and cut inference FLOPs to about 27% of V3.2. The Pro model requires 3x less compute than its predecessor; Flash requires 10x less. In practice this matters a lot. Running Flash locally against models with standard attention at large context windows is a noticeably different experience — the compressed attention keeps things usable where vanilla attention implementations fall apart. ## Training details DeepSeek V4 uses the **Muon optimizer** — the same optimizer Kimi K2 uses, with Kimi's scaling recipe for LLM training. There's an interesting cross-pollination here: Kimi K2 uses DeepSeek-V3 architectural components, while DeepSeek V4 adopts Kimi's optimizer approach. The architecture also introduces **Manifold-Constrained Hyper-Connections** for better signal propagation through the network. All three reasoning effort modes (non-think, think high, think max) are supported in both models. ## How it performs on real tasks A benchmark against Claude Opus 4.7 and Kimi K2.6 using a complex workflow orchestration spec (20 endpoints, lease management, retries, event streaming) gave: - **Claude Opus 4.7**: 91/100 - **DeepSeek V4 Pro**: 77/100 at $2.25 - **Kimi K2.6**: 68/100 - **DeepSeek V4 Flash**: 60/100 at $0.02 Pro's bugs were in lease expiry enforcement, parallel scheduling, and TypeScript build integrity. Flash had a broken route prefix preventing workflow creation and shared the expired-lease bug, but showed surprisingly solid tool-calling behavior. The $0.02 cost per attempt for Flash is the number worth sitting with. For tasks that tolerate imperfect first passes — or where you're running many attempts — the economics are genuinely different from anything else available. ## vLLM support vLLM now supports both V4 models. The implementation handles the heterogeneous attention types (different layers use different attention mechanisms) through a unified 256-token logical block size, compressor state managed as sliding-window KV, and page-size bucketing into three pools. Three kernel fusions deliver 1.4–20x speedups depending on the operation, with multi-stream parallelism adding another 5–6% latency reduction. ## Local inference with DS4 Antirez built a specialized inference engine for DeepSeek V4 Flash called DS4, built on top of llama.cpp and GGML. It runs on an M3 Max 128GB at mixed 2-bit/4-bit quantization (4-bit for the last six layers). A 50-minute demo shows it building a small programming language in real time — not accelerated. One observation from running it: the small KV cache means storing and loading sessions from disk actually works well. On fast SSD storage like a MacBook's, the assumption that disk is a bad target for KV cache doesn't hold anymore. Upcoming work on DS4 targets flatter prefill as context grows, which matters most on DGX Spark hardware. DS4 also includes activation steering — extracting concept vectors from model activations and boosting them during inference. This has been theoretically interesting for years but practically inaccessible because it requires model weights. With a capable open-weight model running locally, it becomes something you can actually experiment with. Whether steering for "unpromptable" concepts or compressing large context into implicit memory turns out to be useful in practice is still an open question. ## The broader picture DeepSeek released strong open base models, skipped the benchmark-optimization theater, and left post-training to whoever wants to pick it up. The open-weight ecosystem now has a 1M-context model with genuinely novel attention architecture at a cost point that changes what's feasible to run. The gap between open and proprietary models on surface coverage is closing; correctness on hard edge cases (lease recovery, cross-run scheduling) is where frontier models still pull ahead. ## Similar posts on daily.dev - [DeepSeek-V4: a million-token context that agents can actually use](https://daily.dev/posts/deepseek-v4-a-million-token-context-that-agents-can-actually-use-uczd4kivx) · Hugging Face · 27 upvotes · 2 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#deepseek](https://daily.dev/tags/deepseek), [#mixture-of-experts](https://daily.dev/tags/mixture-of-experts) [View this post on daily.dev](https://daily.dev/posts/deepseek-v4-preview-two-moe-models-with-1m-context-and-new-attention-mechanism-pbpwljmj3) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"DeepSeek-V4 preview: two MoE models with 1M context and new attention mechanism","url":"https://daily.dev/posts/deepseek-v4-preview-two-moe-models-with-1m-context-and-new-attention-mechanism-pbpwljmj3","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/deepseek-v4-preview-two-moe-models-with-1m-context-and-new-attention-mechanism-pbpwljmj3"},"datePublished":"2026-04-25T00:03:13.813Z","dateModified":"2026-05-22T08:43:16.119Z","description":"DeepSeek released two new open-weight models under the V4 banner: V4-Pro (1.6T parameters, 49B activated) and V4-Flash (284B, 13B activated), both supporting...","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/deepseek-v4-preview-two-moe-models-with-1m-context-and-new-attention-mechanism-pbpwljmj3","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,deepseek,mixture-of-experts","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"DeepSeek-V4 preview: two MoE models with 1M context and new attention mechanism"}]} ```