--- title: Run LLM Locally: 125B Qwen3.8-Flash-Next on 12 GB VRAM description: Strata is an open-source MIT-licensed inference engine, built partly on llama.cpp/ggml, that runs the 125B-parameter Qwen3.8-Flash-Next model on a gaming PC... canonical: https://daily.dev/posts/run-llm-locally-125b-qwen3-8-flash-next-on-12-gb-vram-384y37aho twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Run LLM Locally: 125B Qwen3.8-Flash-Next on 12 GB VRAM | daily.dev og:description: Strata is an open-source MIT-licensed inference engine, built partly on llama.cpp/ggml, that runs the 125B-parameter Qwen3.8-Flash-Next model on a gaming PC... og:url: https://daily.dev/posts/run-llm-locally-125b-qwen3-8-flash-next-on-12-gb-vram-384y37aho og:image: https://api.daily.dev/og/posts/384Y37Aho.png og:image:alt: Run LLM Locally: 125B Qwen3.8-Flash-Next on 12 GB VRAM og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Run LLM Locally: 125B Qwen3.8-Flash-Next on 12 GB VRAM **[FireUp](https://daily.dev/sources/fireup-pro)** · 4 min read · 0 upvotes · 0 comments ## Summary Strata is an open-source MIT-licensed inference engine, built partly on llama.cpp/ggml, that runs the 125B-parameter Qwen3.8-Flash-Next model on a gaming PC with just 12 GB VRAM and 64 GB RAM, hitting 80–94 tokens/s on an RTX 5070. It achieves this through a mixture-of-experts architecture (24,576 small experts, 10 active per token), tiered GPU/RAM/CPU/SSD memory, aggressive 2-4 bit GGUF quantization, and speculative decoding. It exposes OpenAI- and Anthropic-compatible local APIs, a browser chat app, and an MCP server, and has passed 16,000 GitHub stars. Limitations include slow model loading, single-request serving by default, and large download sizes, making it best suited for individual offline development rather than team deployment. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I run a 125B parameter model like Qwen3.8-Flash-Next on a 12 GB GPU? Strata, an MIT-licensed inference engine built partly on llama.cpp/ggml, makes this possible through a mixture-of-experts architecture where only 10 of 24,576 experts activate per token, tiered memory across GPU, RAM, CPU and SSD, 2-4 bit GGUF quantization variants, and speculative decoding. On an RTX 5070 with 12 GB VRAM and 64 GB RAM, it reaches 80-94 tokens per second. _daily.dev surfaces practical setups like this for developers testing local LLM inference on consumer hardware._ ### Which Strata quantization variant should I use for running Qwen3.8-Flash-Next locally? The choice depends on available RAM: 32 GB RAM suits the Coder variant (half the experts removed but keeps 91% of the full model's SWE-bench Verified score), 48 GB suits IQ2_XS or Q2_0, 64 GB suits IQ2_XS as the sweet spot or IQ3_XXS/IQ3_S, and 96 GB+ suits IQ3_S or UD-IQ4_XS for results closest to the full model. _Developers weighing local model variants against hardware constraints can track these comparisons on daily.dev._ ### What are the limitations of running Strata's local LLM setup on a gaming PC? Loading the model takes 1-3 minutes and 35-55 GB of RAM, potentially freezing the PC, the download is about 70 GB, and by default only one request is served at a time, making it suited for a personal workstation rather than a team server. The first message in a long chat processes slowly, around 1 minute per 30,000 tokens, and AMD cards can't read images yet on Windows. _daily.dev helps developers weighing local-first AI setups stay aware of real-world constraints like these before committing._ ## Similar posts on daily.dev - [AI On Your Gaming PC](https://daily.dev/posts/ai-on-your-gaming-pc-a7iadecz7) · Hackaday · 2 upvotes · 2 comments - [My 8GB GPU shouldn't run flagship local LLMs, but this workflow makes it work anyway](https://daily.dev/posts/my-8gb-gpu-shouldn-t-run-flagship-local-llms-but-this-workflow-makes-it-work-anyway-nb2r0gq97) · XDA Developers · 1 upvotes · 0 comments - [Running local models on an M4 with 24GB memory](https://daily.dev/posts/running-local-models-on-an-m4-with-24gb-memory-iezpeho6j) · Hacker News · 4 upvotes · 0 comments - [Local LLMs Are Getting Easier: The Complete Guide \(2026\)](https://daily.dev/posts/local-llms-are-getting-easier-the-complete-guide-2026--gaggzz9j2) · SitePoint · 0 upvotes · 0 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#local-ai](https://daily.dev/tags/local-ai), [#qwen](https://daily.dev/tags/qwen), [#mixture-of-experts](https://daily.dev/tags/mixture-of-experts), [#llama-cpp](https://daily.dev/tags/llama-cpp) [View this post on daily.dev](https://daily.dev/posts/run-llm-locally-125b-qwen3-8-flash-next-on-12-gb-vram-384y37aho) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Run LLM Locally: 125B Qwen3.8-Flash-Next on 12 GB VRAM","url":"https://daily.dev/posts/run-llm-locally-125b-qwen3-8-flash-next-on-12-gb-vram-384y37aho","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/run-llm-locally-125b-qwen3-8-flash-next-on-12-gb-vram-384y37aho"},"datePublished":"2026-10-09T08:16:59.802Z","dateModified":"2026-10-09T09:51:59.854Z","description":"Strata is an open-source MIT-licensed inference engine, built partly on llama.cpp/ggml, that runs the 125B-parameter Qwen3.8-Flash-Next model on a gaming PC...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4e8f32d382d3138de0ca8be46c7c2380?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4e8f32d382d3138de0ca8be46c7c2380?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"FireUp","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"FireUp","logo":"https://media.daily.dev/image/upload/s--JWZsm71i--/f_auto,q_auto/v1774960866/logos/fireup-pro?_a=BAMAMiWQ0","url":"https://daily.dev/sources/fireup-pro"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/run-llm-locally-125b-qwen3-8-flash-next-on-12-gb-vram-384y37aho","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-science,local-ai,qwen,mixture-of-experts,llama-cpp","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"FireUp","item":"https://daily.dev/sources/fireup-pro"},{"@type":"ListItem","position":3,"name":"Run LLM Locally: 125B Qwen3.8-Flash-Next on 12 GB VRAM"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/run-llm-locally-125b-qwen3-8-flash-next-on-12-gb-vram-384y37aho#faq","mainEntity":[{"@type":"Question","name":"How can I run a 125B parameter model like Qwen3.8-Flash-Next on a 12 GB GPU?","acceptedAnswer":{"@type":"Answer","text":"Strata, an MIT-licensed inference engine built partly on llama.cpp/ggml, makes this possible through a mixture-of-experts architecture where only 10 of 24,576 experts activate per token, tiered memory across GPU, RAM, CPU and SSD, 2-4 bit GGUF quantization variants, and speculative decoding. On an RTX 5070 with 12 GB VRAM and 64 GB RAM, it reaches 80-94 tokens per second. daily.dev surfaces practical setups like this for developers testing local LLM inference on consumer hardware."}},{"@type":"Question","name":"Which Strata quantization variant should I use for running Qwen3.8-Flash-Next locally?","acceptedAnswer":{"@type":"Answer","text":"The choice depends on available RAM: 32 GB RAM suits the Coder variant (half the experts removed but keeps 91% of the full model's SWE-bench Verified score), 48 GB suits IQ2_XS or Q2_0, 64 GB suits IQ2_XS as the sweet spot or IQ3_XXS/IQ3_S, and 96 GB+ suits IQ3_S or UD-IQ4_XS for results closest to the full model. Developers weighing local model variants against hardware constraints can track these comparisons on daily.dev."}},{"@type":"Question","name":"What are the limitations of running Strata's local LLM setup on a gaming PC?","acceptedAnswer":{"@type":"Answer","text":"Loading the model takes 1-3 minutes and 35-55 GB of RAM, potentially freezing the PC, the download is about 70 GB, and by default only one request is served at a time, making it suited for a personal workstation rather than a team server. The first message in a long chat processes slowly, around 1 minute per 30,000 tokens, and AMD cards can't read images yet on Windows. daily.dev helps developers weighing local-first AI setups stay aware of real-world constraints like these before committing."}}]} ```