--- title: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU description: Soup is an open-source Python CLI tool that simplifies LLM fine-tuning and post-training into a single config file and one command. It supports SFT, DPO, ORPO,... canonical: https://daily.dev/posts/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu-sa081iua4 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU | daily.dev og:description: Soup is an open-source Python CLI tool that simplifies LLM fine-tuning and post-training into a single config file and one command. It supports SFT, DPO, ORPO,... og:url: https://daily.dev/posts/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu-sa081iua4 og:image: https://api.daily.dev/og/posts/Sa081IuA4.png og:image:alt: Show HN: Fine-tune an 8B model on a 4 GB laptop GPU og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Show HN: Fine-tune an 8B model on a 4 GB laptop GPU **[Hacker News](https://daily.dev/sources/hn)** · 13 min read · 1 upvotes · 1 comments ## Summary Soup is an open-source Python CLI tool that simplifies LLM fine-tuning and post-training into a single config file and one command. It supports SFT, DPO, ORPO, SimPO, KTO, and other training methods with automatic batch size, GPU detection, and quantization. A key feature is layer streaming, which enables training 8B models on GPUs with as little as 4 GB VRAM by streaming frozen base model layers from RAM or NVMe one at a time. The latest v0.72.4 extends layer streaming to preference-based losses (DPO, ORPO, SimPO, KTO). It also includes tools for evaluation, serving (OpenAI-compatible API), export (GGUF, ONNX, TensorRT), and compliance workflows. Works locally without cloud or SSH. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does DPO fine-tuning get a reference model without doubling GPU memory usage when using layer streaming? The same streamed base model is reused with its adapters switched off, avoiding a second copy of the weights. On an RTX 3050 with 4GB VRAM, streamed DPO peaked at 0.914x the memory used by supervised fine-tuning, while forcing a genuine second model added 730MB, exactly one weight copy. The cost instead appears in time: DPO reads the layer stack 1.52x as often per step as SFT. _Anyone budgeting VRAM for preference tuning can track techniques like this via daily.dev._ ### Is KTO actually a reference-free preference optimization method like ORPO and SimPO? No, KTO is not reference-free despite often being described that way; it selects its reference the same way DPO does. ORPO and SimPO genuinely are reference-free. This distinction matters for memory planning during layer-streamed fine-tuning, since KTO needs the same reference-handling treatment as DPO rather than skipping it like ORPO or SimPO. _Developers comparing DPO, KTO, ORPO, and SimPO can follow these nuances on daily.dev._ ### What caused fine-tuned LoRA adapters to silently load as the untuned base model in Soup v0.72.0? Adapter tensors were saved under keys containing an extra '.inner.' segment, so every loader returned the untuned base model instead of the tuned adapter. This was fixed in v0.72.1; anyone who trained on v0.72.0 needs to re-run or re-save their adapter, and can check for the bug by inspecting safetensors keys for '.inner.' substrings. _Teams tracking fine-tuning tool regressions like this stay ahead of breakage via daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 24 comments across hackernews (as of 2026-09-13). **TL;DR:** Commenters are broadly impressed by the layer-streaming technique that enables 8B model fine-tuning on a 4 GB GPU, and appreciate the author's transparent benchmarking and willingness to engage; minor concerns center on speed overhead, documentation gaps, and some early AI-written replies from the author. **Sentiment:** 55% positive · 35% mixed · 10% skeptical **The case for** - The layer-streaming approach is technically clever — keeping the frozen base in host RAM and prefetching one decoder layer at a time genuinely reduces peak VRAM to a single layer's worth. - The author publishes full measurement records, including discarded runs, and enforces bit-exactness against a resident reference as a CI test rather than a one-off claim. - Real-world ROI use cases (e.g., fine-tuned 4B for AML compliance at community banks) validate the small-local-model premise. - The author is responsive and candid, correcting documentation bugs (broken example configs) and admitting mistakes (early LLM-written replies) in real time. **The pushback** - Streaming incurs a measured 1.43× speed overhead versus resident training, which the author acknowledges but cannot eliminate. - The author initially used an LLM to write their own replies, raising authenticity concerns before they corrected course. - Example configs in the repo pointed at tiny test fixtures as training data and used an outdated schema — a documentation/quality gap spotted by a prominent commenter. - The website copy ('Get Started for Free') implies a future paid tier without clarifying terms, and the site's color contrast is reportedly poor. - There is still a hard VRAM floor dependent on model size (e.g., 14B NF4 needs ~7.5 GB page-locked RAM), so the technique doesn't fully eliminate hardware constraints. **By community** - hackernews (positive): Mostly receptive to the engineering approach, with substantive back-and-forth on correctness methodology, dataset sizing, and hardware trade-offs, tempered by mild criticism over the author's early AI-written replies and documentation gaps. **Hottest debate:** Whether the author's initial use of an LLM to write their own forum replies undermines the credibility of the project's claims. **Open questions** - How does Soup auto-tune hyperparameters and make complex training decisions like reward-hacking correction mid-run? - What are the practical dataset size guidelines for different task types and model sizes? - Will the tool remain free, and what does 'Get Started for Free' actually imply about future pricing? **Highlights** > Author here. The constraint everyone works around is that the frozen base has to fit in VRAM. But during LoRA the base is frozen — read, never written. It doesn't need to live in VRAM, it needs to arrive before the matmul that uses it. So it sits in host RAM and streams into a small pool of pre-allocated VRAM buffers, one decoder layer at a time, prefetched one ahead on a dedicated CUDA stream. Peak VRAM becomes one layer instead of the whole model. Measured on an RTX 3050 Laptop (4 GB, Windows): Llama-3.1-8B in NF4 at 119.6 tok/s, 3.32 GB peak, 100% SM occupancy. Also Qwen2.5-3B with an un-quantized bf16 base at 143 tok/s in 2.15 GB, which is CUDA OOM when trained resident on the same card. Overhead is 1.43x vs resident, measured at 0.5B — the only size on this card with a valid resident baseline, and I publish that baseline so you can check the division. Most of the work wasn't speed, it was correctness. Streaming fails silently: cut the autograd path and the loss still falls because the upper layers keep learning. So the bar was bit-exactness against a resident reference of the same numerics — max abs logit difference 0.0, across nine architecture families in two precisions, as a CI test rather than a one-off. That protocol caught a PEFT dispatch defect producing 0.94 logit divergence with byte-identical weights and adapters, no crash, no warning. Not claiming anything above 8B — 14B NF4 needs ~7.5 GB page-locked against a measured 7.12 GB ceiling here, so I didn't run it. All numbers are Windows, so pessimistic vs Linux. Measurement records, including the ones I threw away: https://github.com/MakazhanAlpamys/Soup/tree/main/benchmarks Write-up: https://doi.org/10.5281/zenodo.21771064 Happy to answer anything about the scheduler or the correctness protocol. > — [MakazhanAlpamys on hackernews · 1 comments](https://news.ycombinator.com/item?id=49166987) > Those are format examples and test fixtures. Five to ten rows each. Not training data. You did spot a real problem though. Eight configs in `examples/configs` pointed at those fixtures as training data. Seven were still on the old schema and would not even parse. I've fixed that, added a README to the folder, and a test that parses every config so it doesn't quietly break again. Dataset size mostly depends on the task. Format or style, a few hundred examples is often enough. A task the model already half knows, usually a few thousand. New facts are often a bad fit for fine-tuning. I'd reach for RAG instead. I haven't measured how this changes with model size, so I'm not going to make up a number. > — [MakazhanAlpamys on hackernews](https://news.ycombinator.com/item?id=49175660) > Depends what you change. Format or style, few hundred rows is often enough. A task the model already half knows, few thousand. New facts is where people waste a week. The model comes back wrong in a new way. Use RAG for facts. > — [MakazhanAlpamys on hackernews](https://news.ycombinator.com/item?id=49172162) > Those are mine. I used an LLM for my replies and that was a bad call, I said so further down. Writing them myself now. > — [MakazhanAlpamys on hackernews](https://news.ycombinator.com/item?id=49171740) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49166984) · 104 points · 24 comments ## Community discussion Top comments from developers on daily.dev. **@agustinbarrientos** · 0 upvotes > A 0.0 logit difference in CI makes me trust the 4 GB benchmark far more than the headline alone. ## Similar posts on daily.dev - [We Got Claude to Fine-Tune an Open Source LLM](https://daily.dev/posts/we-got-claude-to-fine-tune-an-open-source-llm-syttcqt2y) · Hugging Face · 4 upvotes · 0 comments - [Fine-tuning and Deploying LLM with Unsloth, SGLang and Runpod](https://daily.dev/posts/fine-tuning-and-deploying-llm-with-unsloth-sglang-and-runpod-ifo5tb1tx) · Daily Dose of Data Science \| Avi Chawla \| Substack · 2 upvotes · 0 comments - [Unsloth and Training Hub: Lightning-fast LoRA and QLoRA fine-tuning](https://daily.dev/posts/unsloth-and-training-hub-lightning-fast-lora-and-qlora-fine-tuning-nhdohkubf) · Red Hat Developer · 1 upvotes · 0 comments - [How Shopify built a continual learning loop with PyTorch and vLLM – PyTorch](https://daily.dev/posts/how-shopify-built-a-continual-learning-loop-with-pytorch-and-vllm-pytorch-viqzi06yu) · PyTorch · 0 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#python](https://daily.dev/tags/python), [#deep-learning](https://daily.dev/tags/deep-learning), [#cli](https://daily.dev/tags/cli) [View this post on daily.dev](https://daily.dev/posts/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu-sa081iua4) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Show HN: Fine-tune an 8B model on a 4 GB laptop GPU","url":"https://daily.dev/posts/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu-sa081iua4","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu-sa081iua4"},"datePublished":"2026-08-04T14:56:50.346Z","dateModified":"2026-09-13T19:30:50.881Z","description":"Soup is an open-source Python CLI tool that simplifies LLM fine-tuning and post-training into a single config file and one command. It supports SFT, DPO, ORPO,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4dc050e9300803c16931d9a29747d871?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4dc050e9300803c16931d9a29747d871?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu-sa081iua4","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"ai,python,deep-learning,cli","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Show HN: Fine-tune an 8B model on a 4 GB laptop GPU"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu-sa081iua4","comment":[{"@type":"Comment","text":"A 0.0 logit difference in CI makes me trust the 4 GB benchmark far more than the headline alone.","datePublished":"2026-08-06T03:33:46.586Z","url":"https://daily.dev/posts/Sa081IuA4#c-m5r95Qbjh","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/show-hn-fine-tune-an-8b-model-on-a-4-gb-laptop-gpu-sa081iua4#faq","mainEntity":[{"@type":"Question","name":"How does DPO fine-tuning get a reference model without doubling GPU memory usage when using layer streaming?","acceptedAnswer":{"@type":"Answer","text":"The same streamed base model is reused with its adapters switched off, avoiding a second copy of the weights. On an RTX 3050 with 4GB VRAM, streamed DPO peaked at 0.914x the memory used by supervised fine-tuning, while forcing a genuine second model added 730MB, exactly one weight copy. The cost instead appears in time: DPO reads the layer stack 1.52x as often per step as SFT. Anyone budgeting VRAM for preference tuning can track techniques like this via daily.dev."}},{"@type":"Question","name":"Is KTO actually a reference-free preference optimization method like ORPO and SimPO?","acceptedAnswer":{"@type":"Answer","text":"No, KTO is not reference-free despite often being described that way; it selects its reference the same way DPO does. ORPO and SimPO genuinely are reference-free. This distinction matters for memory planning during layer-streamed fine-tuning, since KTO needs the same reference-handling treatment as DPO rather than skipping it like ORPO or SimPO. Developers comparing DPO, KTO, ORPO, and SimPO can follow these nuances on daily.dev."}},{"@type":"Question","name":"What caused fine-tuned LoRA adapters to silently load as the untuned base model in Soup v0.72.0?","acceptedAnswer":{"@type":"Answer","text":"Adapter tensors were saved under keys containing an extra '.inner.' segment, so every loader returned the untuned base model instead of the tuned adapter. This was fixed in v0.72.1; anyone who trained on v0.72.0 needs to re-run or re-save their adapter, and can check for the bug by inspecting safetensors keys for '.inner.' substrings. Teams tracking fine-tuning tool regressions like this stay ahead of breakage via daily.dev."}}]} ```