--- title: TorchSpec: Speculative Decoding Training at Scale – PyTorch description: TorchSpec is a PyTorch-native framework for training speculative decoding draft models at scale using a disaggregated architecture. It separates the inference... canonical: https://daily.dev/posts/torchspec-speculative-decoding-training-at-scale-pytorch-selmrhtw4 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: TorchSpec: Speculative Decoding Training at Scale – PyTorch | daily.dev og:description: TorchSpec is a PyTorch-native framework for training speculative decoding draft models at scale using a disaggregated architecture. It separates the inference... og:url: https://daily.dev/posts/torchspec-speculative-decoding-training-at-scale-pytorch-selmrhtw4 og:image: https://api.daily.dev/og/posts/sELmrHTw4.png og:image:alt: TorchSpec: Speculative Decoding Training at Scale – PyTorch og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # TorchSpec: Speculative Decoding Training at Scale – PyTorch **[PyTorch](https://daily.dev/sources/pytorch)** · 10 min read · 0 upvotes · 0 comments ## Summary TorchSpec is a PyTorch-native framework for training speculative decoding draft models at scale using a disaggregated architecture. It separates the inference system (which generates hidden states from a large target model) from the training system (which trains the draft model), streaming tensor data between them via RDMA or TCP using the Mooncake transfer engine. This eliminates the two main bottlenecks of existing approaches: massive disk storage requirements from offline precomputation and GPU memory pressure from co-located inference and training. Using TorchSpec, the team trained a Kimi K2.5 EAGLE-3 draft model on 600k samples (6B tokens) with 1500 H200 GPU hours, achieving over 60% throughput improvement at batch size 1. The framework supports vLLM and SGLang inference engines, long-context sequences up to 200K tokens, and is open-sourced along with the trained draft model and dataset. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [EAGLE 3.1: Advancing Speculative Decoding Through Collaboration Between the EAGLE Team, vLLM, and TorchSpec](https://daily.dev/posts/eagle-3-1-advancing-speculative-decoding-through-collaboration-between-the-eagle-team-vllm-and-to-ognrhklsb) · vLLM · 0 upvotes · 0 comments - [Speculative Decoding: Lossless Acceleration by Guessing, Then Verifying](https://daily.dev/posts/speculative-decoding-lossless-acceleration-by-guessing-then-verifying-oofvgctbs) · Medium · 0 upvotes · 0 comments - [The Machine Learning Practitioner’s Guide to Speculative Decoding](https://daily.dev/posts/the-machine-learning-practitioner-s-guide-to-speculative-decoding-3z4bou5te) · Machine Learning Mastery · 2 upvotes · 0 comments - [Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference](https://daily.dev/posts/co-designing-ai-models-using-speculative-decoding-for-faster-llm-inference-0zr9mkfau) · NVIDIA Developer · 0 upvotes · 0 comments - [How speculative decoding delivers faster LLM inference](https://daily.dev/posts/how-speculative-decoding-delivers-faster-llm-inference-jxnu7qqsj) · Red Hat Developer · 1 upvotes · 0 comments --- Tags: [#ai-inference](https://daily.dev/tags/ai-inference), [#llm](https://daily.dev/tags/llm), [#machine-learning](https://daily.dev/tags/machine-learning), [#pytorch](https://daily.dev/tags/pytorch), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/torchspec-speculative-decoding-training-at-scale-pytorch-selmrhtw4) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"TorchSpec: Speculative Decoding Training at Scale – PyTorch","url":"https://daily.dev/posts/torchspec-speculative-decoding-training-at-scale-pytorch-selmrhtw4","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/torchspec-speculative-decoding-training-at-scale-pytorch-selmrhtw4"},"datePublished":"2026-03-19T22:30:51.212Z","dateModified":"2026-10-14T02:26:55.903Z","description":"TorchSpec is a PyTorch-native framework for training speculative decoding draft models at scale using a disaggregated architecture. It separates the inference...","image":"https://media.daily.dev/image/upload/s--foaA6JGU--/f_auto/v1722860399/public/Placeholder%2004","thumbnailUrl":"https://media.daily.dev/image/upload/s--foaA6JGU--/f_auto/v1722860399/public/Placeholder%2004","isAccessibleForFree":true,"articleSection":"PyTorch","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"PyTorch","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/5fdab3d75a834a97a52d433d9c7e5ff9","url":"https://daily.dev/sources/pytorch"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/torchspec-speculative-decoding-training-at-scale-pytorch-selmrhtw4","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-inference,llm,machine-learning,pytorch,vllm","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"PyTorch","item":"https://daily.dev/sources/pytorch"},{"@type":"ListItem","position":3,"name":"TorchSpec: Speculative Decoding Training at Scale – PyTorch"}]} ```