--- title: DeepSeek open-sources DeepInfer, its in-house inference... description: DeepSeek has open-sourced DeepInfer, the in-house inference framework it uses to serve DeepSeek-V4.1-Flash at scale, releasing it under the Apache 2.0 license.... canonical: https://daily.dev/posts/deepseek-open-sources-deepinfer-its-in-house-inference-framework-8gu6kds8a twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: DeepSeek open-sources DeepInfer, its in-house inference framework | daily.dev og:description: DeepSeek has open-sourced DeepInfer, the in-house inference framework it uses to serve DeepSeek-V4.1-Flash at scale, releasing it under the Apache 2.0 license.... og:url: https://daily.dev/posts/deepseek-open-sources-deepinfer-its-in-house-inference-framework-8gu6kds8a og:image: https://api.daily.dev/og/posts/8gU6KdS8a.png og:image:alt: DeepSeek open-sources DeepInfer, its in-house inference framework og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # DeepSeek open-sources DeepInfer, its in-house inference framework **[Collections](https://daily.dev/sources/collections)** · 3 min read · 0 upvotes · 0 comments ## Summary DeepSeek has open-sourced DeepInfer, the in-house inference framework it uses to serve DeepSeek-V4.1-Flash at scale, releasing it under the Apache 2.0 license. The framework was initially built primarily on vLLM, and vLLM acknowledged the collaboration, noting it has invested heavily in inference efficiency for DeepSeek V4.1 Flash and plans to incorporate ideas from DeepInfer going forward. ## Content DeepSeek has open-sourced DeepInfer, the inference framework it uses for large-scale production serving, under the Apache 2.0 license. It had previously said a full release wasn't practical. The GitHub repo passed 6,300 stars within eight hours and more than 8,000 by Tuesday afternoon. ## What it supports DeepInfer currently runs only DeepSeek-V4.1-Flash. Hardware support is Nvidia's SM100 (Blackwell) GPUs and Huawei's Ascend 950 NPUs, both from day one. Older Nvidia chips such as Hopper are not supported. Batch-invariant inference and deterministic sampling work only on Nvidia hardware. Building it requires Linux, GCC 13+, Python 3.12, and either CUDA 13 or CANN 9.2. ## What's in it The release is the speed layer: - Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect. - Parallelism: attention data parallelism, balanced context parallelism, and MoE expert parallelism with EPLB. - DSpark speculative decoding, which drafts up to 5 tokens ahead and uses confidence estimates to schedule verification. - Bounded replay, which trims prefill by recomputing only a 140-token suffix sized to cover the sliding attention window. - Asynchronous batch scheduling and structured output constraints (regex, JSON schema, EBNF). A companion library, deepseek-recipe, is planned. It will convert Chat Completions, Responses and Messages requests into DeepSeek V4 and V4.1 prompts matching the official API's behavior. ## What's missing This is offline inference, not a full serving stack. A complete production service still needs an external KV cache system, prefill/decode disaggregation, and load balancing across replicas. ## Built on vLLM DeepInfer was built mainly on vLLM, with ideas from SGLang and TensorRT-LLM. The vLLM project thanked DeepSeek for the release and for crediting the open-source community, and said it will keep improving V4.1 Flash efficiency with ideas from DeepInfer. ## The numbers DeepSeek's engineering report claims a single cluster serves 155 trillion tokens a day, with a 96% input-cache hit rate, 400ms time to first token and 230 tokens/s generation. Its benchmarks used four SM100 GPUs for prompts and 64 for output. DeepSeek reports $38 of revenue per accelerator-hour at off-peak pricing. Analysts estimate that implies inference margins above 80% and an 81-day payback at scale. vLLM says it came within 6% of DeepInfer's published decode throughput on GB200 hardware. That was with Wide-EP and EPLB, DeepSeek's decode configuration (V4.1-Flash, 131K context, nextn=3) and batch sizes 8 to 256. On the AgentX workload with GB300, vLLM reports up to 1M tokens/s/GPU and $38.96 per GPU-hour at 196 P90 TPS, using DeepSeek's off-peak API prices. It also reports up to 826 TPS per user with extended low-latency kernel support. ## Reaction Thom Wolf listed DeepSeek's earlier releases (FlashMLA, DeepEP, DeepGEMM, the 3FS file system) and called the streak "absurd." @antirez compared DeepInfer to his DwarfStar project: a vertical inference engine for one model on specific hardware. He called it a good move and a lesson for those who criticized the approach. I find the Huawei detail the most interesting part. Ascend 950 sits next to Blackwell as a first-class target, not an afterthought. ## Questions this post answers ### What is DeepInfer and who built it? DeepInfer is DeepSeek's in-house inference framework, now released as fully open source under the Apache 2.0 license. It was initially built primarily on vLLM and powers DeepSeek-V4.1-Flash at scale. DeepSeek credited the open-source community in the release, and the vLLM project publicly thanked DeepSeek for the contribution. _Teams evaluating LLM inference stacks follow framework releases like this one on daily.dev._ ### Is DeepInfer based on vLLM? Yes, DeepInfer was initially built primarily on vLLM, according to the vLLM project itself. vLLM stated it has invested significant effort into inference efficiency specifically for DeepSeek-V4.1-Flash and plans to continue improving performance by drawing on ideas introduced in DeepInfer. _Anyone comparing vLLM-based inference stacks can track these updates on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 213 comments across hackernews (as of 2026-10-21). **TL;DR:** Discussion centers less on the technical release itself and more on its geopolitical and economic implications — whether open-sourcing DeepSeek's inference stack undermines the moat and valuations of US frontier labs, with heavy debate on profitability, enterprise adoption of Chinese models, and motives behind the release. **Sentiment:** 40% positive · 35% mixed · 25% skeptical **The case for** - Many argue this accelerates commoditization of inference, driving down costs and benefiting users/enterprises regardless of who built it. - Some see it as evidence that inference is highly profitable, with cited figures suggesting strong margins even at off-peak pricing. - Several commenters say enterprises (including in Europe) are already considering or adopting Chinese open models for cost and sovereignty reasons. **The pushback** - Skeptics question whether big corporations will actually trust and deploy Chinese-origin models due to security, regulatory, and adversarial-payload concerns. - Some doubt the release meaningfully threatens OpenAI/Anthropic's moat, noting those labs' advantages still include enterprise contracts, hardware access, and data. - Others suspect strategic/geopolitical motives behind the open-sourcing (undermining US AI valuations) rather than pure altruism or open-source spirit. **By community** - hackernews (heated): Debate sprawls across geopolitics, whether this erodes US AI labs' moats, inference profitability economics, and skepticism about enterprise adoption of Chinese models, with sharp disagreement throughout. **Hottest debate:** Whether open-sourcing this stack genuinely threatens the valuations/moats of OpenAI and Anthropic, or whether enterprise security and regulatory concerns will keep Chinese-origin models out of serious business use regardless. **Open questions** - Why would DeepSeek give up trade secrets that seem to keep getting absorbed by competitors like Anthropic/OpenAI? - Does the quoted $38/card/hour figure represent real profit after accounting for full infrastructure costs, or just revenue? - Will regulatory/security concerns actually stop large enterprises from adopting open Chinese models at scale? **Highlights** > I've been trying to find an answer to that, too. The best I've got right now is this: Moat-like things going for them: - access to the US government (for huge inflated government contracts as a source of free capital, having the competition banned, having their products legally mandated, getting other preferential treatment, having foreign governments bullied into granting similar advantages) Too unsustainable, too fragile, too small: - Strong brands: Brand loyalty seems like a pretty insignificant factor in AI right now - Semi-exclusive hardware access, RAM buyouts: Unsustainable even in the short term - Lock-in effects: Their models aren't very stable across versions, or very unique, and switching providers is about as easy as it can be in a highly competitive market. - Exclusive technology that is important, advanced and hard to replicate to a degree that would materially set them apart and justify potentially much higher cost. Maybe I'm missing something here, but a lot of the claims they make to that end fall apart even on light scrutiny (due to things like test sets that are small, hand-picked and clearly overfitted on, or broad general claims that cannot be substantiated). I'm getting the feeling they don't have anything that materially sets them apart at all, but that's seems impossible to measure in any meaningful way, so who knows, they might have something here. > — [dividedbyzero on hackernews](https://news.ycombinator.com/item?id=50164580) > A nuclear bomb "Based on this framework, we have achieved high-throughput, low-latency large-scale inference services for DeepSeek V4.1 Flash. Currently, the service throughput of a single cluster is 155 trillion tokens/day, and the overall revenue is $38 per card per hour (at off-peak pricing)" Source: https://zhuanlan.zhihu.com/p/2095560295930508953 > — [schneehertz on hackernews · 6 comments](https://news.ycombinator.com/item?id=50163494) > This isn't true, especially not with the inference engine becoming public, now you can build your on models on the same or similar architecture and ship massive gains on top of these stacks. I think people don't realize the Salesforce SAP Palantir would love to ship these models, and they have/are shipping them. The threat of exfilteration of data via these models is high but it's nowhere near as high as the margins. For instance if you could cut your ai input bill by over 90% would you do it? Can you put some of that extra margin say 10-20% to make it more secure. I know of atleast 10 fortune 500 companies that are entirely planning to or have moved to Chinese models for a large portion of their work. It's insane people in finance are delusional enough to think that won't happen. Maybe Govt will try to de-risk with bans on using open models but I don't think that's going to work at scale, people will literally spawn labs to launder DeepSeek/GLM models into a US model by some novel training etc approach or something. I still don't think that it's as much a kami-kaze rather it's an attempt to normalize the world around the Chinese stack. If DeepInfer is supported by everyone, it will be much easier to export stuff aka hardware that only supports it for now or is optimized for it. Assume china starts selling hyper optimized Asics for inference at some point. This feels like a race to the bottom as soon as possible because they have much lower costs, if there is a Wipeout in AI Labs in US chinese labs are largely insulated, and they can hopefully buy Nvidia chips for cheap if that does happen. > — [minraws on hackernews](https://news.ycombinator.com/item?id=50164528) > I'm wondering how it benefits Deepseek. Why would they willingly give up their trade secrets? It's obviously very valuable, as they are running a profitable company. It follows the same trend as when they released their cache optimizations a few months back: why do it? What followed their release of this technology was that Anthropic and OpenAI silently released models which were on-par or similarly intelligent, but "surprisingly" had much cheaper cached tokens. So their inventions were obviously just worked into their competitors, who just got much greater margins + became more profitable. The same thing goes here: I'm sure there are inventions and optimizations which are unknown to Deepseeks competitors which yet again will be worked in. So once again, they're letting their trade secrets go. My question is: why? How does Deepseek benefit from this? There must be some gain I'm not seeing as they keep doing this play over and over, and I doubt it's "for the good of open source", as China is practically as isolated as you can get. To believe that this is in the spirit of the open source community is heavily naive. > — [baalimago on hackernews · 3 comments](https://news.ycombinator.com/item?id=50163908) > I saw the same kind of comments when the first Deepseek model was initially released. However the relevance that investors assign to something like this does not seem to match the excitement expressed here. I think they do not believe that big corporations in the US or Europe would actually use chinese models in their business operations due to security, regulatory and/or business continuity concerns. > — [bonplan23 on hackernews · 11 comments](https://news.ycombinator.com/item?id=50164066) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50163277) · 524 points · 213 comments ## Similar posts on daily.dev - [DeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0](https://daily.dev/posts/deepseek-v4-1-flash-on-vllm-5x-agentic-throughput-since-day-0-js5lna7tm) · vLLM · 0 upvotes · 0 comments --- Tags: [#open-source](https://daily.dev/tags/open-source), [#llm](https://daily.dev/tags/llm), [#ai-inference](https://daily.dev/tags/ai-inference), [#deepseek](https://daily.dev/tags/deepseek), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/deepseek-open-sources-deepinfer-its-in-house-inference-framework-8gu6kds8a) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"DeepSeek open-sources DeepInfer, its in-house inference framework","url":"https://daily.dev/posts/deepseek-open-sources-deepinfer-its-in-house-inference-framework-8gu6kds8a","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/deepseek-open-sources-deepinfer-its-in-house-inference-framework-8gu6kds8a"},"datePublished":"2026-10-20T09:47:30.201Z","dateModified":"2026-10-21T23:52:17.367Z","description":"DeepSeek has open-sourced DeepInfer, the in-house inference framework it uses to serve DeepSeek-V4.1-Flash at scale, releasing it under the Apache 2.0 license....","image":"https://pbs.twimg.com/media/HVEJ3G7XoAE-vSF.png","thumbnailUrl":"https://pbs.twimg.com/media/HVEJ3G7XoAE-vSF.png","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/deepseek-open-sources-deepinfer-its-in-house-inference-framework-8gu6kds8a","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"open-source,llm,ai-inference,deepseek,vllm","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"DeepSeek open-sources DeepInfer, its in-house inference framework"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/deepseek-open-sources-deepinfer-its-in-house-inference-framework-8gu6kds8a#faq","mainEntity":[{"@type":"Question","name":"What is DeepInfer and who built it?","acceptedAnswer":{"@type":"Answer","text":"DeepInfer is DeepSeek's in-house inference framework, now released as fully open source under the Apache 2.0 license. It was initially built primarily on vLLM and powers DeepSeek-V4.1-Flash at scale. DeepSeek credited the open-source community in the release, and the vLLM project publicly thanked DeepSeek for the contribution. Teams evaluating LLM inference stacks follow framework releases like this one on daily.dev."}},{"@type":"Question","name":"Is DeepInfer based on vLLM?","acceptedAnswer":{"@type":"Answer","text":"Yes, DeepInfer was initially built primarily on vLLM, according to the vLLM project itself. vLLM stated it has invested significant effort into inference efficiency specifically for DeepSeek-V4.1-Flash and plans to continue improving performance by drawing on ideas introduced in DeepInfer. Anyone comparing vLLM-based inference stacks can track these updates on daily.dev."}}]} ```