--- title: A new way of TRAINING LLMs with 3x less memory...!? description: A new research paper called 'Diffusion Blocks' from Sakana AI reframes transformer training as a diffusion denoising process, allowing the network to be split... canonical: https://daily.dev/posts/a-new-way-of-training-llms-with-3x-less-memory--qetjubjls twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: A new way of TRAINING LLMs with 3x less memory...!? | daily.dev og:description: A new research paper called 'Diffusion Blocks' from Sakana AI reframes transformer training as a diffusion denoising process, allowing the network to be split... og:url: https://daily.dev/posts/a-new-way-of-training-llms-with-3x-less-memory--qetjubjls og:image: https://api.daily.dev/og/posts/qeTJUbjLs.png og:image:alt: A new way of TRAINING LLMs with 3x less memory...!? og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # A new way of TRAINING LLMs with 3x less memory...!? **[bycloud](https://daily.dev/sources/bycloud)** · 16 min read · 0 upvotes · 0 comments ## Summary A new research paper called 'Diffusion Blocks' from Sakana AI reframes transformer training as a diffusion denoising process, allowing the network to be split into independently trainable blocks instead of requiring full end-to-end backpropagation. This could reduce training memory usage by roughly the number of blocks used (potentially 3-6x, theoretically more), while also cutting cross-GPU communication overhead during distributed training. Tests so far are limited to toy models (50M-few hundred million parameters) across vision transformers, diffusion transformers, masked diffusion language models, and autoregressive language models, showing comparable or even slightly improved performance versus standard training. However, ablations show diminishing returns with too many blocks, and the method has not yet been validated at billion-parameter scale. The video also includes a sponsored segment for a GPU cloud provider. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the diffusion blocks method from Sakana AI and how does it reduce LLM training memory? Diffusion blocks reframes transformer training as a diffusion denoising process, exploiting the residual connections between layers. The transformer is split into blocks (e.g., 12 layers into 3 or 4 blocks), each assigned a noise range, and each block is trained independently without backpropagating through the whole network, cutting memory use roughly by a factor equal to the number of blocks (B), so 3 blocks gives about 3x less memory and 6 blocks up to 6x. _Engineers exploring cheaper LLM training setups can follow emerging memory-saving techniques like this via daily.dev._ ### What model scale has the diffusion blocks training method been tested on so far? Testing so far covers only toy-scale models ranging from about 50 million to a few hundred million parameters, including a 12-layer ViT for image classification, DiT models for image generation, masked diffusion language models, and a 12-layer LLaMA-2-style autoregressive transformer split into four blocks. It has not been confirmed to work at billion-parameter scale, so its real-world impact on large LLM training remains unproven. _Track whether experimental training techniques like this scale to production LLMs by following AI research updates on daily.dev._ ### Why doesn't a training memory-saving technique like diffusion blocks also reduce memory needed for LLM inference? Because inference still requires the model to run causally as one connected unit, so the entire model must remain in GPU memory to generate output quickly. Offloading to CPU is already possible but trades latency for memory, which isn't worthwhile for inference speed, so diffusion blocks' benefits apply mainly to training and research, not to serving models in production. _Developers weighing training versus inference memory trade-offs can keep up with such distinctions on daily.dev._ ## Similar posts on daily.dev - [The Anatomy of Diffusion LLMs](https://daily.dev/posts/the-anatomy-of-diffusion-llms-41s7k4vg9) · Daily Dose of Data Science \| Avi Chawla \| Substack · 1 upvotes · 0 comments - [Beyond Standard LLMs](https://daily.dev/posts/beyond-standard-llms-b1iaquuim) · Sebastian Raschka · 28 upvotes · 1 comments --- Tags: [#gpu](https://daily.dev/tags/gpu), [#diffusion-models](https://daily.dev/tags/diffusion-models) [View this post on daily.dev](https://daily.dev/posts/a-new-way-of-training-llms-with-3x-less-memory--qetjubjls) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"A new way of TRAINING LLMs with 3x less memory...!?","url":"https://daily.dev/posts/a-new-way-of-training-llms-with-3x-less-memory--qetjubjls","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/a-new-way-of-training-llms-with-3x-less-memory--qetjubjls"},"datePublished":"2026-08-18T06:14:47.121Z","dateModified":"2026-09-14T08:57:24.399Z","description":"A new research paper called 'Diffusion Blocks' from Sakana AI reframes transformer training as a diffusion denoising process, allowing the network to be split...","image":"https://i.ytimg.com/vi/T_GhB7lK2YE/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/T_GhB7lK2YE/sddefault.jpg","isAccessibleForFree":true,"articleSection":"bycloud","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"bycloud","logo":"https://media.daily.dev/image/upload/s--UUgAibgb--/f_auto/v1712315122/logos/bycloud","url":"https://daily.dev/sources/bycloud"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/a-new-way-of-training-llms-with-3x-less-memory--qetjubjls","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"gpu,diffusion-models","timeRequired":"PT16M","video":{"@type":"VideoObject","name":"A new way of TRAINING LLMs with 3x less memory...!?","description":"A new research paper called 'Diffusion Blocks' from Sakana AI reframes transformer training as a diffusion denoising process, allowing the network to be split...","thumbnailUrl":"https://i.ytimg.com/vi/T_GhB7lK2YE/sddefault.jpg","uploadDate":"2026-08-18T06:14:47.121Z","duration":"PT16M","url":"https://api.daily.dev/r/qeTJUbjLs","embedUrl":"https://www.youtube.com/embed/T_GhB7lK2YE"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"bycloud","item":"https://daily.dev/sources/bycloud"},{"@type":"ListItem","position":3,"name":"A new way of TRAINING LLMs with 3x less memory...!?"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/a-new-way-of-training-llms-with-3x-less-memory--qetjubjls#faq","mainEntity":[{"@type":"Question","name":"What is the diffusion blocks method from Sakana AI and how does it reduce LLM training memory?","acceptedAnswer":{"@type":"Answer","text":"Diffusion blocks reframes transformer training as a diffusion denoising process, exploiting the residual connections between layers. The transformer is split into blocks (e.g., 12 layers into 3 or 4 blocks), each assigned a noise range, and each block is trained independently without backpropagating through the whole network, cutting memory use roughly by a factor equal to the number of blocks (B), so 3 blocks gives about 3x less memory and 6 blocks up to 6x. Engineers exploring cheaper LLM training setups can follow emerging memory-saving techniques like this via daily.dev."}},{"@type":"Question","name":"What model scale has the diffusion blocks training method been tested on so far?","acceptedAnswer":{"@type":"Answer","text":"Testing so far covers only toy-scale models ranging from about 50 million to a few hundred million parameters, including a 12-layer ViT for image classification, DiT models for image generation, masked diffusion language models, and a 12-layer LLaMA-2-style autoregressive transformer split into four blocks. It has not been confirmed to work at billion-parameter scale, so its real-world impact on large LLM training remains unproven. Track whether experimental training techniques like this scale to production LLMs by following AI research updates on daily.dev."}},{"@type":"Question","name":"Why doesn't a training memory-saving technique like diffusion blocks also reduce memory needed for LLM inference?","acceptedAnswer":{"@type":"Answer","text":"Because inference still requires the model to run causally as one connected unit, so the entire model must remain in GPU memory to generate output quickly. Offloading to CPU is already possible but trades latency for memory, which isn't worthwhile for inference speed, so diffusion blocks' benefits apply mainly to training and research, not to serving models in production. Developers weighing training versus inference memory trade-offs can keep up with such distinctions on daily.dev."}}]} ```