--- title: Build and Train a 25M Parameter LLM From Scratch on Your CPU description: A free hour-long freeCodeCamp.org YouTube tutorial walks through building, pre-training, and fine-tuning a 25-million parameter language model entirely on a... canonical: https://daily.dev/posts/build-and-train-a-25m-parameter-llm-from-scratch-on-your-cpu-hpb2ytmzl twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Build and Train a 25M Parameter LLM From Scratch on Your CPU | daily.dev og:description: A free hour-long freeCodeCamp.org YouTube tutorial walks through building, pre-training, and fine-tuning a 25-million parameter language model entirely on a... og:url: https://daily.dev/posts/build-and-train-a-25m-parameter-llm-from-scratch-on-your-cpu-hpb2ytmzl og:image: https://api.daily.dev/og/posts/HpB2Ytmzl.png og:image:alt: Build and Train a 25M Parameter LLM From Scratch on Your CPU og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Build and Train a 25M Parameter LLM From Scratch on Your CPU **[freeCodeCamp](https://daily.dev/sources/freecodecamp)** · 2 min read · 121 upvotes · 4 comments ## Summary A free hour-long freeCodeCamp.org YouTube tutorial walks through building, pre-training, and fine-tuning a 25-million parameter language model entirely on a CPU, without needing a GPU cluster. It covers modern LLM architecture concepts like hybrid linear/sparse attention, Mixture of Experts, tied embedding weights, and multimodal image patch inputs, then moves through pre-training from random weights using cross-entropy loss, and post-training via a reinforcement learning loop where the model writes code, runs against unit tests, and earns rewards, raising task success from 0% to passing most evaluations. It also frames the process like real AI research: forming hypotheses, tuning temperature and reward design, and checking statistical significance across multiple seeds. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Can you train a language model on a CPU instead of a GPU? Yes, a 25-million parameter language model can be built, pre-trained, and fine-tuned entirely on a CPU using a compact byte-level vocabulary that keeps training loops fast enough to run locally in seconds. This scaled-down size preserves the same core architecture and math used by frontier models, just without the expensive cloud compute costs. _daily.dev surfaces resources like this for developers exploring LLM training without GPU access._ ### How can reinforcement learning improve a code-generating language model's success rate? A lightweight RL loop has the model write code, run it against automated unit tests, and receive rewards based on results, which can push task success rates from 0% up to passing the majority of evaluations. The process also involves tuning temperature and reward structures and validating improvements across multiple random seeds for statistical significance. _Developers tuning RL-based training loops can track techniques like this via daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@harshitmanepally** · 7 upvotes > Free code camp the goat **@alexx\_dev** · 1 upvotes > training a 25 M model on a single CPU will push memory bandwidth and take weeks, what batching/quantization strategy are you using also, without fp16 or int8 the RAM usage will explode quickly **@ristotoldsep** · 1 upvotes > Training a tiny LLM on CPU sounds useless right up until you actually understand the stack. love this. **@yeon21** · 1 upvotes > it takes more than 45mins to cover a build & train architecture even when most of the working parts are glossed over, more like 112hrs imho. ## Similar posts on daily.dev - [Learn how to fine-tune LLMs in 12 hours](https://daily.dev/posts/learn-how-to-fine-tune-llms-in-12-hours-8n7yittla) · freeCodeCamp · 116 upvotes · 1 comments - [LLM From Scratch is a hands-on workshop where you write every piece of an AI from nothing](https://daily.dev/posts/llm-from-scratch-is-a-hands-on-workshop-where-you-write-every-piece-of-an-ai-from-nothing-ntyfh6hef) · XDA Developers · 0 upvotes · 0 comments - [Writing an LLM from scratch, part 22 -- finally training our LLM\!](https://daily.dev/posts/writing-an-llm-from-scratch-part-22----finally-training-our-llm--f31twqx5a) · Hacker News · 8 upvotes · 0 comments - [Pretraining a Llama Model on Your Local GPU](https://daily.dev/posts/pretraining-a-llama-model-on-your-local-gpu-4jub4kfdq) · Machine Learning Mastery · 3 upvotes · 0 comments - [An LLM From “Scratch”](https://daily.dev/posts/an-llm-from-scratch--liyi2mnpd) · Hackaday · 1 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#python](https://daily.dev/tags/python), [#llm](https://daily.dev/tags/llm), [#reinforcement-learning](https://daily.dev/tags/reinforcement-learning), [#mixture-of-experts](https://daily.dev/tags/mixture-of-experts) [View this post on daily.dev](https://daily.dev/posts/build-and-train-a-25m-parameter-llm-from-scratch-on-your-cpu-hpb2ytmzl) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Build and Train a 25M Parameter LLM From Scratch on Your CPU","url":"https://daily.dev/posts/build-and-train-a-25m-parameter-llm-from-scratch-on-your-cpu-hpb2ytmzl","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/build-and-train-a-25m-parameter-llm-from-scratch-on-your-cpu-hpb2ytmzl"},"datePublished":"2026-10-08T04:03:33.660Z","dateModified":"2026-10-08T05:42:21.146Z","description":"A free hour-long freeCodeCamp.org YouTube tutorial walks through building, pre-training, and fine-tuning a 25-million parameter language model entirely on a...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/730d4d0016f6598939873ec34ae3fde5?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/730d4d0016f6598939873ec34ae3fde5?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"freeCodeCamp","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"freeCodeCamp","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1628412854/logos/freecodecamp","url":"https://daily.dev/sources/freecodecamp"},"commentCount":4,"discussionUrl":"https://daily.dev/posts/build-and-train-a-25m-parameter-llm-from-scratch-on-your-cpu-hpb2ytmzl","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":121},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":4}],"keywords":"machine-learning,python,llm,reinforcement-learning,mixture-of-experts","timeRequired":"PT2M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"freeCodeCamp","item":"https://daily.dev/sources/freecodecamp"},{"@type":"ListItem","position":3,"name":"Build and Train a 25M Parameter LLM From Scratch on Your CPU"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/build-and-train-a-25m-parameter-llm-from-scratch-on-your-cpu-hpb2ytmzl","comment":[{"@type":"Comment","text":"Free code camp the goat","datePublished":"2026-10-12T05:38:14.658Z","url":"https://daily.dev/posts/HpB2Ytmzl#c-JDNlAHKEz","author":{"@type":"Person","name":"HARSHIT MANEPALLY","url":"https://daily.dev/harshitmanepally","image":"https://media.daily.dev/image/upload/s--O0TOmw4y--/f_auto/v1715772965/public/noProfile"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":7}},{"@type":"Comment","text":"training a 25 M model on a single CPU will push memory bandwidth and take weeks, what batching/quantization strategy are you using also, without fp16 or int8 the RAM usage will explode quickly","datePublished":"2026-10-12T15:10:50.382Z","url":"https://daily.dev/posts/HpB2Ytmzl#c-l4DbdQ9XF","author":{"@type":"Person","name":"alex","url":"https://daily.dev/alexx_dev"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"Training a tiny LLM on CPU sounds useless right up until you actually understand the stack. love this.","datePublished":"2026-10-13T11:50:58.600Z","url":"https://daily.dev/posts/HpB2Ytmzl#c-Kioj7dZpJ","author":{"@type":"Person","name":"Risto Tõldsep","url":"https://daily.dev/ristotoldsep","image":"https://lh3.googleusercontent.com/a/ACg8ocLDWc6mZn0JwNmXXw6WY0L_HJ6pegzRttooC5VgtXESHSj3MNYx=s96-c"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"it takes more than 45mins to cover a build & train architecture even when most of the working parts are glossed over, more like 112hrs imho.","datePublished":"2026-10-14T10:33:09.002Z","url":"https://daily.dev/posts/HpB2Ytmzl#c-rXUXJt5sD","author":{"@type":"Person","name":"Yeon","url":"https://daily.dev/yeon21","image":"https://media.daily.dev/image/upload/s--0y--U8hL--/f_auto/v1781177930/avatars/avatar_ytFXfSuvulhE7N6ULXRcK?_a=BAMAMiWQ0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/build-and-train-a-25m-parameter-llm-from-scratch-on-your-cpu-hpb2ytmzl#faq","mainEntity":[{"@type":"Question","name":"Can you train a language model on a CPU instead of a GPU?","acceptedAnswer":{"@type":"Answer","text":"Yes, a 25-million parameter language model can be built, pre-trained, and fine-tuned entirely on a CPU using a compact byte-level vocabulary that keeps training loops fast enough to run locally in seconds. This scaled-down size preserves the same core architecture and math used by frontier models, just without the expensive cloud compute costs. daily.dev surfaces resources like this for developers exploring LLM training without GPU access."}},{"@type":"Question","name":"How can reinforcement learning improve a code-generating language model's success rate?","acceptedAnswer":{"@type":"Answer","text":"A lightweight RL loop has the model write code, run it against automated unit tests, and receive rewards based on results, which can push task success rates from 0% up to passing the majority of evaluations. The process also involves tuning temperature and reward structures and validating improvements across multiple random seeds for statistical significance. Developers tuning RL-based training loops can track techniques like this via daily.dev."}}]} ```