--- title: GitHub - sqliteai/waste: Run the full... description: WASTE (Weight-Aware Streaming Tensor Engine) is a dependency-free, embeddable C inference engine that runs the full 2.78-trillion-parameter Kimi K3 model on a... canonical: https://daily.dev/posts/github---sqliteai-waste-run-the-full-2-78-trillion-parameter-kimi-k3-model-beyond-available-ram-by--o7y68bmgf twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine. | daily.dev og:description: WASTE (Weight-Aware Streaming Tensor Engine) is a dependency-free, embeddable C inference engine that runs the full 2.78-trillion-parameter Kimi K3 model on a... og:url: https://daily.dev/posts/github---sqliteai-waste-run-the-full-2-78-trillion-parameter-kimi-k3-model-beyond-available-ram-by--o7y68bmgf og:image: https://api.daily.dev/og/posts/o7y68bmgF.png og:image:alt: GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine. **[Hacker News](https://daily.dev/sources/hn)** · 29 min read · 0 upvotes · 0 comments ## Summary WASTE (Weight-Aware Streaming Tensor Engine) is a dependency-free, embeddable C inference engine that runs the full 2.78-trillion-parameter Kimi K3 model on a consumer laptop by streaming expert weights directly from NVMe storage. Instead of requiring the full 982 GB model in RAM, it keeps the model trunk resident (~27 GB) and streams only the activated experts per token from disk, using remaining RAM as a bounded expert cache. On a 64 GB MacBook Pro M5 Pro, it achieves 0.49–0.54 tokens/second. The engine uses 3-bit residual vector quantization for experts, bypasses the OS page cache for direct NVMe reads, and implements an absorbed KV cache for K3's hybrid attention. It also supports multimodal inputs (images via a 27-layer ViT), an OpenAI-compatible HTTP server, and runs on macOS, Linux, and Windows. The project documents all measured optimizations including those that were tried and rejected, with detailed performance tables and engineering rationale. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community take How the wider developer community reacted, aggregated from 4 discussions and 95 comments across hackernews (as of 2026-08-01). **TL;DR:** The community is split between skepticism about the LLM-generated README/codebase and genuine technical curiosity about running Kimi K3 in 29GB RAM; the 0.5 tok/s speed is widely seen as a major practical limitation, while the LLM-authored code and docs draw significant criticism. **Sentiment:** 20% positive · 40% mixed · 40% skeptical **The case for** - Transparency about AI authorship is appreciated by some, who prefer it over concealment. - The project demonstrates a novel approach to SSD-offloaded inference with custom NEON ARM optimizations that outperformed Metal on macOS. - Slow overnight inference could still be useful for batch tasks like code review or summarization run while the machine is idle. - The cost per million tokens (~$5) is calculable and the hardware capex is far lower than a GPU cluster. **The pushback** - The README is widely identified as LLM-generated slop — verbose, internally inconsistent, and written for an audience with full project context rather than external readers. - At 0.5 tok/s, the model is too slow for most practical interactive use cases. - The headline 29GB RAM claim is questioned as potentially contradicting what is known about Kimi K3's actual parameter footprint. - The company 'sqliteai' is criticized for riding on the SQLite name and having a history of non-open licenses. - No pre-converted weights are available; users must download and convert the full model themselves. **By community** - hackernews (mixed): Commenters are genuinely interested in the technical feat but heavily critical of the LLM-generated documentation, the practical usability at 0.5 tok/s, and the credibility of the headline memory claims. **Hottest debate:** Whether letting an LLM author commits, READMEs, and code is acceptable laziness or a legitimate and transparent new workflow. **Open questions** - Is the 29GB RAM figure accurate given Kimi K3's known dense parameter size, and what exactly is being quantized/omitted? - How does this compare to llama.cpp's native SSD mmap offloading, and what is the concrete benefit of a custom implementation? - What are the long-term SSD wear implications of streaming weights at this rate continuously? - Will pre-converted .waste weights be made available for download? **Highlights** > Yeah I'm begging these authors to at least *read* the LLM generated README's. They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development. These decisions and instructions from the user are extremely important for the model and almost completely irrelevant for an outside reader looking at a "finished" product. So, we get sentences like this: "Where the levers were is not where they are. Overlapping the expert reads with the arithmetic was worth ~1.6x and shipped; the two that looked bigger — reading fewer bytes per token, and keeping more of them in RAM — were both measured and both refused, one because this family's router has no tail to demote and one because a cache the machine will not leave resident cannot be bought at any price." What the fuck does that mean? Obviously some internal development decision, using the absolutely inscrutable internal terminology that Claude loves. If people would just read what they publish, I'm sure this would stick out immediately. I'm not an LLM hater, I use them a ton and they work very well for writing complex code, it's undeniable. But they generate absolute dogshit first draft writing. > — [bglazer on hackernews · 2 comments](https://news.ycombinator.com/item?id=49127408) > Yup, I hate to engage in anything that looks like a "shallow dismissal" but the project documentation seems to outright contradict itself wrt. whether it's running the model at genuinely native precision (though the claimed 3-bit quant is potentially interesting) and the headline claim of achieving 2 secs/token in a mere 29GB RAM footprint looks outright nonsensical given what we know about K3 itself (~115GB in dense parameters alone at native precision, plus ~25GB active sparse experts per token and some comparatively minor footprint for the KV cache). This is just not very helpful. > — [zozbot234 on hackernews](https://news.ycombinator.com/item?id=49128572) > Yes, I do this all the time, and also check in the co-authored project plans which drove the commits. For a project that is transparently only possible due to agentic coding, I don't see any reason to conceal the methods. > — [bensyverson on hackernews](https://news.ycombinator.com/item?id=49126702) > I have one setup that gets about 1.5 tok/s of a very large on prem LLM. It's used for overnight project review runs and code review that it is fed at the end of each work day. When I look at it the next morning it has done quite a lot of useful work. Dealing with a big slow LLM as an effective tool is really about planning the workflow to feed it. > — [walrus01 on hackernews](https://news.ycombinator.com/item?id=49130646) > So if this Mac uses 30-50W, that's 40-60 tok/Wh...vs maybe 80k for a modern GPU cluster? So that's about 1000-2000x more power for the SSD streaming, unfortunately. > — [herf on hackernews · 1 comments](https://news.ycombinator.com/item?id=49127044) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49123386) · 51 points · 91 comments - [hackernews](https://news.ycombinator.com/item?id=49106591) · 9 points · 4 comments - [hackernews](https://news.ycombinator.com/item?id=49112587) · 2 points · 0 comments - [hackernews](https://news.ycombinator.com/item?id=49098395) · 8 points · 0 comments ## Similar posts on daily.dev - [Kimi K3 runs on 8GB CPU, No GPU](https://daily.dev/posts/kimi-k3-runs-on-8gb-cpu-no-gpu-khbh5h19l) · Medium · 0 upvotes · 0 comments --- Tags: [#c](https://daily.dev/tags/c), [#local-ai](https://daily.dev/tags/local-ai), [#ai-inference](https://daily.dev/tags/ai-inference), [#mixture-of-experts](https://daily.dev/tags/mixture-of-experts), [#kimi-k3](https://daily.dev/tags/kimi-k3) [View this post on daily.dev](https://daily.dev/posts/github---sqliteai-waste-run-the-full-2-78-trillion-parameter-kimi-k3-model-beyond-available-ram-by--o7y68bmgf) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.","url":"https://daily.dev/posts/github---sqliteai-waste-run-the-full-2-78-trillion-parameter-kimi-k3-model-beyond-available-ram-by--o7y68bmgf","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/github---sqliteai-waste-run-the-full-2-78-trillion-parameter-kimi-k3-model-beyond-available-ram-by--o7y68bmgf"},"datePublished":"2026-07-31T18:25:09.119Z","dateModified":"2026-08-01T04:42:00.203Z","description":"WASTE (Weight-Aware Streaming Tensor Engine) is a dependency-free, embeddable C inference engine that runs the full 2.78-trillion-parameter Kimi K3 model on a...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9f7b5fc660f86257da82dff36e5946d8?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9f7b5fc660f86257da82dff36e5946d8?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/github---sqliteai-waste-run-the-full-2-78-trillion-parameter-kimi-k3-model-beyond-available-ram-by--o7y68bmgf","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"c,local-ai,ai-inference,mixture-of-experts,kimi-k3","timeRequired":"PT29M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine."}]} ```