--- title: Stop Guessing Which Local Model To Run | daily.dev description: Running coding agents against local models fails not because hardware is inadequate but because people pick the wrong model/quantization/context configuration... canonical: https://daily.dev/posts/stop-guessing-which-local-model-to-run-li3wod384 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Stop Guessing Which Local Model To Run | daily.dev og:description: Running coding agents against local models fails not because hardware is inadequate but because people pick the wrong model/quantization/context configuration... og:url: https://daily.dev/posts/stop-guessing-which-local-model-to-run-li3wod384 og:image: https://api.daily.dev/og/posts/li3wOD384.png og:image:alt: Stop Guessing Which Local Model To Run og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Stop Guessing Which Local Model To Run **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 12 min read · 1 upvotes · 0 comments ## Summary Running coding agents against local models fails not because hardware is inadequate but because people pick the wrong model/quantization/context configuration for their machine. Agent workloads differ fundamentally from chat: conversations accumulate tokens across many turns, tool calls demand high precision (no room for quantization degradation), and speed must stay high throughout a long loop rather than just feel responsive in a chat window. Memory bandwidth, not compute, caps token generation speed, and this bandwidth is rarely published or comparable across machines. Magnitude, an open-source inference server, profiles a machine's chip, memory, and real-world bandwidth, then recommends and auto-tunes a full configuration (model, quantization, context size, speculative decoding, concurrency) and wires it into existing coding agent harnesses like Claude Code, Codex, Cline, and OpenCode via a single CLI command. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does a coding agent get slower and start making mistakes when running on a local model, when chat felt fine? Agent trajectories accumulate conversation history across 20-30 turns, including tool output and file contents, which occupies RAM alongside model weights and can exceed the weights' own size. Chat sessions stay a few thousand tokens, but agent loops compound slow speeds and demand high precision for tool calls, so quantization degradation that's tolerable in chat causes structural failures in agent tool calling. _Developers debugging local-agent slowdowns can dig into hardware and inference tradeoffs on daily.dev._ ### How much memory does Qwen3.6 35B-A3B need to run locally at 8-bit versus 4-bit quantization? At 8-bit, Qwen3.6 35B-A3B's weights take roughly 38GB, fitting on a 64GB machine with room for a long conversation, while full BF16 weights run around 70GB and won't fit at all. At 4-bit, weights still total about 20GB, which exceeds a 16GB machine's capacity before adding any conversation context, since all experts must stay resident regardless of per-token routing. _Anyone sizing local models against their hardware can track these tradeoffs on daily.dev._ ### What does memory bandwidth have to do with local LLM inference speed? Generation speed is capped roughly by memory bandwidth divided by the size of the weights being read, since producing each token requires reading the model's weights out of memory. Compute barely matters at this stage, so a high-throughput GPU won't help if the model is too large for its onboard memory and spills into slower system memory, and achieved bandwidth often differs from theoretical specs due to architecture and thermal behavior. _Engineers picking hardware for local inference can follow bandwidth and model-sizing guidance on daily.dev._ ## Similar posts on daily.dev - [Experiences with local models for coding](https://daily.dev/posts/experiences-with-local-models-for-coding-jggvqgz1l) · Martin Fowler · 4 upvotes · 0 comments - [Viability of local models for coding](https://daily.dev/posts/viability-of-local-models-for-coding-1cgbfrvir) · Martin Fowler · 0 upvotes · 0 comments - [Running local models on an M4 with 24GB memory](https://daily.dev/posts/running-local-models-on-an-m4-with-24gb-memory-iezpeho6j) · Hacker News · 4 upvotes · 0 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#ai-agents](https://daily.dev/tags/ai-agents), [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/stop-guessing-which-local-model-to-run-li3wod384) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Stop Guessing Which Local Model To Run","url":"https://daily.dev/posts/stop-guessing-which-local-model-to-run-li3wod384","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/stop-guessing-which-local-model-to-run-li3wod384"},"datePublished":"2026-09-02T19:12:43.856Z","dateModified":"2026-09-02T20:42:45.238Z","description":"Running coding agents against local models fails not because hardware is inadequate but because people pick the wrong model/quantization/context configuration...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/67509c7f1fd36ce32c7461558d7079c5?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/67509c7f1fd36ce32c7461558d7079c5?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Daily Dose of Data Science | Avi Chawla | Substack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Daily Dose of Data Science | Avi Chawla | Substack","logo":"https://media.daily.dev/image/upload/s--4IHQgTOw--/f_auto/v1710503712/logos/dailydoseofds","url":"https://daily.dev/sources/dailydoseofds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/stop-guessing-which-local-model-to-run-li3wod384","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-science,ai-agents,ai-inference","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Daily Dose of Data Science | Avi Chawla | Substack","item":"https://daily.dev/sources/dailydoseofds"},{"@type":"ListItem","position":3,"name":"Stop Guessing Which Local Model To Run"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/stop-guessing-which-local-model-to-run-li3wod384#faq","mainEntity":[{"@type":"Question","name":"Why does a coding agent get slower and start making mistakes when running on a local model, when chat felt fine?","acceptedAnswer":{"@type":"Answer","text":"Agent trajectories accumulate conversation history across 20-30 turns, including tool output and file contents, which occupies RAM alongside model weights and can exceed the weights' own size. Chat sessions stay a few thousand tokens, but agent loops compound slow speeds and demand high precision for tool calls, so quantization degradation that's tolerable in chat causes structural failures in agent tool calling. Developers debugging local-agent slowdowns can dig into hardware and inference tradeoffs on daily.dev."}},{"@type":"Question","name":"How much memory does Qwen3.6 35B-A3B need to run locally at 8-bit versus 4-bit quantization?","acceptedAnswer":{"@type":"Answer","text":"At 8-bit, Qwen3.6 35B-A3B's weights take roughly 38GB, fitting on a 64GB machine with room for a long conversation, while full BF16 weights run around 70GB and won't fit at all. At 4-bit, weights still total about 20GB, which exceeds a 16GB machine's capacity before adding any conversation context, since all experts must stay resident regardless of per-token routing. Anyone sizing local models against their hardware can track these tradeoffs on daily.dev."}},{"@type":"Question","name":"What does memory bandwidth have to do with local LLM inference speed?","acceptedAnswer":{"@type":"Answer","text":"Generation speed is capped roughly by memory bandwidth divided by the size of the weights being read, since producing each token requires reading the model's weights out of memory. Compute barely matters at this stage, so a high-throughput GPU won't help if the model is too large for its onboard memory and spills into slower system memory, and achieved bandwidth often differs from theoretical specs due to architecture and thermal behavior. Engineers picking hardware for local inference can follow bandwidth and model-sizing guidance on daily.dev."}}]} ```