--- title: GitHub - lyogavin/airllm: AirLLM 70B inference with... description: AirLLM is an open-source Python library that enables running massive LLMs on consumer-grade GPUs by streaming one model layer at a time, keeping only that... canonical: https://daily.dev/posts/github---lyogavin-airllm-airllm-70b-inference-with-single-4gb-gpu-s8ukkihfl twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: GitHub - lyogavin/airllm: AirLLM 70B inference with single 4GB GPU | daily.dev og:description: AirLLM is an open-source Python library that enables running massive LLMs on consumer-grade GPUs by streaming one model layer at a time, keeping only that... og:url: https://daily.dev/posts/github---lyogavin-airllm-airllm-70b-inference-with-single-4gb-gpu-s8ukkihfl og:image: https://api.daily.dev/og/posts/s8ukKihFL.png og:image:alt: GitHub - lyogavin/airllm: AirLLM 70B inference with single 4GB GPU og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # GitHub - lyogavin/airllm: AirLLM 70B inference with single 4GB GPU **[Hacker News](https://daily.dev/sources/hn)** · 9 min read · 3 upvotes · 1 comments ## Summary AirLLM is an open-source Python library that enables running massive LLMs on consumer-grade GPUs by streaming one model layer at a time, keeping only that layer in VRAM. This approach allows 70B models to run on 4GB GPUs, 405B Llama 3.1 on 8GB, DeepSeek-V3 (671B) on ~12GB, and even the 2.8T Kimi K3 on under 4GB — all without quantization, distillation, or pruning. The library supports virtually all major open LLM families (Llama, Qwen, DeepSeek, Mistral, Phi, Gemma, etc.) via a single AutoModel API, and also offers optional block-wise quantization for up to 3x inference speedup with minimal accuracy loss. MacOS with Apple Silicon is supported via MLX. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much GPU VRAM does AirLLM need to run Kimi K3 and what dependencies are required? AirLLM runs the 2.8T-parameter Kimi K3 model in 3.72GB of VRAM, measured end to end on a single RTX 6000 Ada, by streaming only the experts a token actually routes to rather than loading whole layers. Running it requires installing compressed-tensors and flash-attn, since Kimi K3's model code mandates flash attention, a CUDA 12 build of torch (no CUDA 13 flash-attn wheel exists yet), and transformers 4.56.x specifically, since its remote code fails to load on 5.x. _Anyone chasing lower VRAM footprints for huge MoE models can follow Kimi K3 and AirLLM updates on daily.dev._ ### How much GPU memory do I need to run DeepSeek-V3 671B or Qwen3-235B locally? DeepSeek-V3 (671B parameters) runs on roughly 12GB of GPU VRAM, and Qwen3-235B-A22B runs on roughly 3GB, using AirLLM's layer-streaming approach that keeps only one layer (or, for MoE models, only the routed experts) on the GPU at a time. VRAM requirements scale with a model's per-layer size rather than its total parameter count, which is why sparse MoE models need dramatically less memory than dense models of similar scale. _Developers sizing hardware for giant open models can track memory-saving techniques like this on daily.dev._ ### How does AirLLM run a 70B parameter model on a 4GB GPU without quantization? AirLLM keeps only one transformer layer loaded on the GPU at any given time, streaming layers from disk or CPU memory as needed, so VRAM usage depends on a single layer's size rather than the model's total size. This lets a 70B Llama model run on 4GB, and optional 4-bit or 8-bit block-wise weight quantization (via bitsandbytes) can additionally speed inference up to 3x with minimal accuracy loss. _Engineers weighing quantization versus layer streaming for constrained GPUs can follow such techniques on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 3 discussions and 76 comments across hackernews (as of 2026-09-13). **TL;DR:** The community is impressed by the engineering ambition but deeply skeptical of practical utility: Kimi K3 runs at 292 seconds per token on a 48GB GPU, making it comically slow for almost any real use case, and many question whether this beats simply running on CPU or using a cloud API. **Sentiment:** 20% positive · 30% mixed · 50% skeptical **The case for** - Layer-by-layer streaming enables running massive models on consumer hardware without quantization, distillation, or pruning. - Useful for non-time-sensitive batch workloads (e.g., overnight processing of private/sensitive data that can't leave local machines). - Represents meaningful engineering progress toward democratizing large model access, even if not yet practical. **The pushback** - 292 seconds per token on a 48GB GPU makes it effectively unusable for interactive or even most batch workloads. - Running the model via cloud API (e.g., Kimi K3's official API) is estimated to be ~80x cheaper and vastly faster. - The project appears vibe-coded and likely to be abandoned, joining a crowded field of similar unmaintained projects. - No clear advantage over existing solutions like llama.cpp with mmap/expert-streaming flags and quantized models. - Documentation and concrete usage examples are reportedly lacking. **By community** - hackernews (skeptical): Commenters are largely amused and critical — the 292 s/token speed is the dominant talking point, with many concluding the tool has almost no practical use case over cloud APIs or llama.cpp. **Hottest debate:** Whether any real-world use case justifies 292 seconds per token when cloud APIs offer the same model orders of magnitude faster and cheaper. **Open questions** - How does this compare in speed and quality to simply running a quantized model via llama.cpp with mmap/expert-streaming on CPU? - Is there a meaningful use case beyond non-time-sensitive, privacy-sensitive batch processing? - Will any of these layer-streaming projects gain enough momentum and maintenance to become a reliable tool? **Highlights** > For anyone wondering “how slow is this?” IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token https://github.com/lyogavin/airllm/releases/tag/v3.1.0 > — [imenani on hackernews · 11 comments](https://news.ycombinator.com/item?id=49155253) > that's 0.003 tokens/second. To get an hour's work done that's normally 30 tokens/second (108k output tokens in an hour) will take 416 days at this rate. And if you're using 100 watts, during that time you will spend $124.61 in electricity, as well as not being able to use your device for something else, plus the noise and heat from your device. For $124, on Moonshot's official Kimi K3 API rates ($0.30 per 1M cached input, $3 per 1M fresh input, $15 per 1M fresh output), you can purchase 42 million fresh-input tokens, or 8.3 million generated output tokens, in whatever mix you want. So what you get is 80x more expensive and you wait 416 days to get it. > — [logicallee on hackernews](https://news.ycombinator.com/item?id=49156087) > If you want it to respond in under a minute, you need more hardware than this application is intended for. This thing's response is measured in seconds per token, not tokens per second. To get Claude Code responsiveness from even a pretty small (but still usable) model, you need, maybe two 32GB GPUs? I run Gemma 4 31B and Qwen 3.6 27B on my dual 32GB GPU setup (cheap old Radeon Pro V620 GPUs) at about 20 t/s, which is not fast enough for comfortable interactive use. A couple of new GPUs, like Radeon AI Pro 9700 at $1400 each, probably gets you fast enough for comfortable interactive use with small models like those. Those small models are not competitive with Claude models, however (maybe they beat Haiku sometimes). They can write a little Python or make a web page, they can't architect a real application. To run a near-frontier model like Kimi K3 or GLM 5.2 at comfortable speeds, you need serious hardware with 768GB VRAM, minimum. I think Asus is releasing something like that for about $150k soon. You can run DeepSeek V4 Flash at almost comfortable speeds and in a decently capable quantization on two DGX Sparks or Asus GX10s (about $10,000). Or, you could use DeepSeek V4 Flash from DeepSeek.com, at blistering speeds and with huge contexts, for something like a decade or two for that same $10,000. > — [SwellJoe on hackernews](https://news.ycombinator.com/item?id=49162964) > > but are they functionally useful or helpful to anyone currently? I've been building a SaaS that deals with data that can't be distributed to third parties. Some of the useful AI stuff I can add is not time sensitive and can run overnight. Things like this allow me to use higher quality models without selling my house for GPUs. > — [dsl on hackernews](https://news.ycombinator.com/item?id=49160910) > I've been seeing a lot of projects that let one use large models on machines with small amounts of memory. They seem to all be doing significant quantization and/or expert streaming. What's the benefit of these projects over something like taking an unsloth quant and running llama.cpp with appropriate flags (-cmoe/-mmap) to manage VRAM vs RAM vs SSD? > — [hatthew on hackernews](https://news.ycombinator.com/item?id=49162451) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49154228) · 187 points · 76 comments - [hackernews](https://news.ycombinator.com/item?id=49142538) · 2 points · 0 comments - [hackernews](https://news.ycombinator.com/item?id=47293735) · 2 points · 0 comments ## Community discussion Top comments from developers on daily.dev. **@matchpoint** · 0 upvotes >  ## Similar posts on daily.dev - [Your old GPU can still run big LLMs – you just need the right tweaks](https://daily.dev/posts/your-old-gpu-can-still-run-big-llms-you-just-need-the-right-tweaks-3sa1xwzxu) · XDA Developers · 0 upvotes · 0 comments - [Guide to Local LLMs in 2026: Privacy, Tools & Hardware](https://daily.dev/posts/guide-to-local-llms-in-2026-privacy-tools-hardware-yigh17bqi) · SitePoint · 1 upvotes · 0 comments - [How to Run LLMs Locally: A Practical Guide for Developers](https://daily.dev/posts/how-to-run-llms-locally-a-practical-guide-for-developers-tijurupob) · BigData Boutique blog · 2 upvotes · 0 comments - [Fast & Efficient LLM Inference with vLLM: A New Course with DeepLearning.AI](https://daily.dev/posts/fast-efficient-llm-inference-with-vllm-a-new-course-with-deeplearning-ai-p35im3usb) · vLLM · 3 upvotes · 0 comments --- Tags: [#genai](https://daily.dev/tags/genai), [#jupyter](https://daily.dev/tags/jupyter) [View this post on daily.dev](https://daily.dev/posts/github---lyogavin-airllm-airllm-70b-inference-with-single-4gb-gpu-s8ukkihfl) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"GitHub - lyogavin/airllm: AirLLM 70B inference with single 4GB GPU","url":"https://daily.dev/posts/github---lyogavin-airllm-airllm-70b-inference-with-single-4gb-gpu-s8ukkihfl","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/github---lyogavin-airllm-airllm-70b-inference-with-single-4gb-gpu-s8ukkihfl"},"datePublished":"2026-08-03T13:48:54.644Z","dateModified":"2026-09-13T19:43:14.362Z","description":"AirLLM is an open-source Python library that enables running massive LLMs on consumer-grade GPUs by streaming one model layer at a time, keeping only that...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/59ade2bb4e45c540cd9ee6c5cf2b2152?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/59ade2bb4e45c540cd9ee6c5cf2b2152?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/github---lyogavin-airllm-airllm-70b-inference-with-single-4gb-gpu-s8ukkihfl","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"genai,jupyter","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"GitHub - lyogavin/airllm: AirLLM 70B inference with single 4GB GPU"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/github---lyogavin-airllm-airllm-70b-inference-with-single-4gb-gpu-s8ukkihfl","comment":[{"@type":"Comment","text":"","datePublished":"2026-08-04T02:37:38.101Z","url":"https://daily.dev/posts/s8ukKihFL#c-MK0yelklA","author":{"@type":"Person","name":"Numb Code","url":"https://daily.dev/matchpoint","image":"https://lh3.googleusercontent.com/a/ACg8ocJ86YIewk804FajHaEbzIBiQa9RFZVR0fs-vBpSZ3ObyDY6Qz8=s96-c"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/github---lyogavin-airllm-airllm-70b-inference-with-single-4gb-gpu-s8ukkihfl#faq","mainEntity":[{"@type":"Question","name":"How much GPU VRAM does AirLLM need to run Kimi K3 and what dependencies are required?","acceptedAnswer":{"@type":"Answer","text":"AirLLM runs the 2.8T-parameter Kimi K3 model in 3.72GB of VRAM, measured end to end on a single RTX 6000 Ada, by streaming only the experts a token actually routes to rather than loading whole layers. Running it requires installing compressed-tensors and flash-attn, since Kimi K3's model code mandates flash attention, a CUDA 12 build of torch (no CUDA 13 flash-attn wheel exists yet), and transformers 4.56.x specifically, since its remote code fails to load on 5.x. Anyone chasing lower VRAM footprints for huge MoE models can follow Kimi K3 and AirLLM updates on daily.dev."}},{"@type":"Question","name":"How much GPU memory do I need to run DeepSeek-V3 671B or Qwen3-235B locally?","acceptedAnswer":{"@type":"Answer","text":"DeepSeek-V3 (671B parameters) runs on roughly 12GB of GPU VRAM, and Qwen3-235B-A22B runs on roughly 3GB, using AirLLM's layer-streaming approach that keeps only one layer (or, for MoE models, only the routed experts) on the GPU at a time. VRAM requirements scale with a model's per-layer size rather than its total parameter count, which is why sparse MoE models need dramatically less memory than dense models of similar scale. Developers sizing hardware for giant open models can track memory-saving techniques like this on daily.dev."}},{"@type":"Question","name":"How does AirLLM run a 70B parameter model on a 4GB GPU without quantization?","acceptedAnswer":{"@type":"Answer","text":"AirLLM keeps only one transformer layer loaded on the GPU at any given time, streaming layers from disk or CPU memory as needed, so VRAM usage depends on a single layer's size rather than the model's total size. This lets a 70B Llama model run on 4GB, and optional 4-bit or 8-bit block-wise weight quantization (via bitsandbytes) can additionally speed inference up to 3x with minimal accuracy loss. Engineers weighing quantization versus layer streaming for constrained GPUs can follow such techniques on daily.dev."}}]} ```