--- title: Janas: Run LLMs larger than RAM by streaming experts... description: Janas-LLM is an experimental, dependency-free C engine for running mixture-of-experts language models (like Qwen3-Next-80B and Gemma 4) on consumer laptops... canonical: https://daily.dev/posts/janas-run-llms-larger-than-ram-by-streaming-experts-from-ssd-sjewosq4e twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Janas: Run LLMs larger than RAM by streaming experts from SSD | daily.dev og:description: Janas-LLM is an experimental, dependency-free C engine for running mixture-of-experts language models (like Qwen3-Next-80B and Gemma 4) on consumer laptops... og:url: https://daily.dev/posts/janas-run-llms-larger-than-ram-by-streaming-experts-from-ssd-sjewosq4e og:image: https://api.daily.dev/og/posts/SjEWOsq4E.png og:image:alt: Janas: Run LLMs larger than RAM by streaming experts from SSD og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Janas: Run LLMs larger than RAM by streaming experts from SSD **[Hacker News](https://daily.dev/sources/hn)** · 17 min read · 0 upvotes · 0 comments ## Summary Janas-LLM is an experimental, dependency-free C engine for running mixture-of-experts language models (like Qwen3-Next-80B and Gemma 4) on consumer laptops with limited RAM by streaming experts from NVMe SSD and caching frequently used ones in memory. Benchmarks on an Intel Core Ultra 9 185H laptop with 32 GB RAM show it outperforming llama.cpp across all 13 tested models, most dramatically on the 48 GB Qwen3-Next-80B model where it reaches 37 tok/s writing versus llama.cpp's 2.8 tok/s, since llama.cpp runs out of memory without special flags. The project includes a benchmarking tool (janas-bench), MCP server support, HTTP API, and code completion integration, and is explicitly labeled an early-stage experiment with unstable API and limited hardware testing. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I run a 80B parameter MoE model like Qwen3-Next-80B on a laptop with only 32GB of RAM? Janas-LLM streams the experts a token needs from an NVMe SSD while keeping frequently reused experts cached in memory, letting a 48.4 GB Qwen3-Next-80B-A3B model run on a 32 GB laptop. On an Intel Core Ultra 9 185H with its integrated Arc GPU, it reached 37 tok/s writing with speculative drafts (24 without) and 62 tok/s reading a prompt, compared to llama.cpp's 2.8 tok/s writing and 16 tok/s reading using mmap with repacking disabled. _For developers pushing large MoE models onto limited hardware, daily.dev surfaces benchmarks and tooling comparisons like this one._ ### Why does llama.cpp run out of memory when running Qwen3-Next-80B with partial GPU offload on a 32GB machine? llama.cpp's CPU backend copies weights into its own memory to repack them, and its Vulkan backend copies the experts left on the CPU into buffers the system cannot reclaim, so even with `-ngl 99 -ncmoe 48` it runs out of memory on a 32 GB laptop. Workarounds include building with `GGML_CPU_REPACK=OFF` and adding `--no-host 1 -lm mmap`, which keep experts in the file and read them through the page cache instead. _Developers tuning llama.cpp memory flags for large MoE models can track configuration trade-offs like these via daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 61 comments across hackernews (as of 2026-10-13). **TL;DR:** Reactions are mixed: some find the SSD-streaming approach a useful trick for running huge MoE models on modest hardware, while others dismiss it as a reinvention of existing techniques (llama.cpp, swap, other similar projects) or question whether the benchmarked speeds are actually usable for real work. **Sentiment:** 30% positive · 40% mixed · 30% skeptical **The case for** - Several see this as part of a broader, encouraging trend of open-weight models and tooling making large models runnable on consumer hardware. - Some report genuinely good practical results from similar expert-streaming approaches, getting tens of tokens/sec on modest consumer GPUs/RAM. - The idea of streaming experts from fast storage is seen as having further headroom, e.g. multi-SSD or networked expert storage. - One commenter notes this approach can deliver up to 2x the performance of llama.cpp on the same weights when optimized for a single model. **The pushback** - Many argue this is just a reinvention of swap/memory-mapping techniques llama.cpp and similar projects already support. - Skepticism that the reported tokens/sec numbers reflect real agentic workloads, since prefill/prompt-processing speed matters far more than generation speed for tool-heavy use. - Some question whether the project is largely vibe-coded, pointing to odd model name hallucinations and documentation gaps. - Concerns about the caching/eviction strategy for experts being non-trivial and not yet a solved problem. - Worry that constant SSD streaming could cause excessive wear on the drive. **By community** - hackernews (heated): Discussion splits between people excited about democratizing local LLM inference and others calling the approach derivative, vibe-coded, or impractical for agentic workloads. **Hottest debate:** Whether streaming experts from SSD is a genuinely useful innovation or just a repeatedly-reinvented version of swap/memory-mapping that llama.cpp already handles. **Open questions** - How well does the caching/eviction algorithm for experts generalize across different models and prompts? - Can this approach work over networked or distributed storage (e.g. S3, multiple nodes) rather than just local NVMe? - What is the real-world impact on SSD wear and longevity from constant expert streaming? **Highlights** > Every few weeks, someone stumbles on what llama.cpp has had for years again, and reimplements it. I'm not sure why. > — [jdb9001 on hackernews · 3 comments](https://news.ycombinator.com/item?id=50066255) > My impression based on a cursory look at the research is that the caching algorithm is absolutely vital, non-obvious, model-dependent, and potentially prompt-dependent to make sure you're holding onto the right experts. It doesn't look like a solved problem yet. > — [Espressosaurus on hackernews](https://news.ycombinator.com/item?id=50067279) > The numbers you povided are for output/generation. Input is around 5-20 thousand for Astra, and 16-60k for Luna. So a few hundred times faster. Most of the tokens consumed by agents is reading stuff - docs, tool outputs, code etc. Previewing a content of a dumb single page of html is a few thousand tokens. If you use claude or some other harness you can ask it how many cache writes/input tokens are there in avg session, and how much time it would take if preprocessing speed was just 60 tok/s. > — [kolinko on hackernews](https://news.ycombinator.com/item?id=50065566) > It lists a bunch of models it supports, including Qwen3.8-Flash > — [DiabloD3 on hackernews · 1 comments](https://news.ycombinator.com/item?id=50066150) > I think this is still the tip of the iceberg in terms of this method. There is no particular requirement for this to be limited to a single SSD. What happens when someone makes a GPU Card with enough SSDs attached to feed every RAM chip individually. > — [Lerc on hackernews](https://news.ycombinator.com/item?id=50068868) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50063717) · 163 points · 61 comments ## Similar posts on daily.dev - [I replaced ChatGPT with a local model on my gaming PC, and it's beating the cloud where I didn't expect](https://daily.dev/posts/i-replaced-chatgpt-with-a-local-model-on-my-gaming-pc-and-it-s-beating-the-cloud-where-i-didn-t-exp-boruivfcs) · XDA Developers · 0 upvotes · 0 comments - [Qwen 3.6 27B is the sweet spot for local development](https://daily.dev/posts/qwen-3-6-27b-is-the-sweet-spot-for-local-development-iqrlzajwn) · Quesma · 2 upvotes · 0 comments - [Top 7 Small Language Models You Can Run on a Laptop](https://daily.dev/posts/top-7-small-language-models-you-can-run-on-a-laptop-uggt1hdi5) · Machine Learning Mastery · 3 upvotes · 0 comments - [Run LLM Locally: 125B Qwen3.8-Flash-Next on 12 GB VRAM](https://daily.dev/posts/run-llm-locally-125b-qwen3-8-flash-next-on-12-gb-vram-384y37aho) · FireUp · 1 upvotes · 0 comments - [Untitled](https://daily.dev/posts/untitled-ptqwtc5dz) · SitePoint · 0 upvotes · 0 comments --- Tags: [#c](https://daily.dev/tags/c) [View this post on daily.dev](https://daily.dev/posts/janas-run-llms-larger-than-ram-by-streaming-experts-from-ssd-sjewosq4e) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Janas: Run LLMs larger than RAM by streaming experts from SSD","url":"https://daily.dev/posts/janas-run-llms-larger-than-ram-by-streaming-experts-from-ssd-sjewosq4e","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/janas-run-llms-larger-than-ram-by-streaming-experts-from-ssd-sjewosq4e"},"datePublished":"2026-10-13T09:08:55.109Z","dateModified":"2026-10-13T12:10:36.053Z","description":"Janas-LLM is an experimental, dependency-free C engine for running mixture-of-experts language models (like Qwen3-Next-80B and Gemma 4) on consumer laptops...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e107f9f9f0005c74a37ce827fbea5ddf?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e107f9f9f0005c74a37ce827fbea5ddf?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/janas-run-llms-larger-than-ram-by-streaming-experts-from-ssd-sjewosq4e","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"c","timeRequired":"PT17M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Janas: Run LLMs larger than RAM by streaming experts from SSD"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/janas-run-llms-larger-than-ram-by-streaming-experts-from-ssd-sjewosq4e#faq","mainEntity":[{"@type":"Question","name":"How can I run a 80B parameter MoE model like Qwen3-Next-80B on a laptop with only 32GB of RAM?","acceptedAnswer":{"@type":"Answer","text":"Janas-LLM streams the experts a token needs from an NVMe SSD while keeping frequently reused experts cached in memory, letting a 48.4 GB Qwen3-Next-80B-A3B model run on a 32 GB laptop. On an Intel Core Ultra 9 185H with its integrated Arc GPU, it reached 37 tok/s writing with speculative drafts (24 without) and 62 tok/s reading a prompt, compared to llama.cpp's 2.8 tok/s writing and 16 tok/s reading using mmap with repacking disabled. For developers pushing large MoE models onto limited hardware, daily.dev surfaces benchmarks and tooling comparisons like this one."}},{"@type":"Question","name":"Why does llama.cpp run out of memory when running Qwen3-Next-80B with partial GPU offload on a 32GB machine?","acceptedAnswer":{"@type":"Answer","text":"llama.cpp's CPU backend copies weights into its own memory to repack them, and its Vulkan backend copies the experts left on the CPU into buffers the system cannot reclaim, so even with `-ngl 99 -ncmoe 48` it runs out of memory on a 32 GB laptop. Workarounds include building with `GGML_CPU_REPACK=OFF` and adding `--no-host 1 -lm mmap`, which keep experts in the file and read them through the page cache instead. Developers tuning llama.cpp memory flags for large MoE models can track configuration trade-offs like these via daily.dev."}}]} ```