--- title: Stanford and Together AI study: local AI efficiency... description: A Stanford and Together AI study introduces an 'intelligence per watt' metric to gauge how efficiently local AI models convert energy into usable capability.... canonical: https://daily.dev/posts/stanford-and-together-ai-study-local-ai-efficiency-improved-5-3-since-2023-but-hard-reasoning-tas-0cuniozjt twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Stanford and Together AI study: local AI efficiency improved 5.3× since 2023, but hard reasoning tasks remain a wall | daily.dev og:description: A Stanford and Together AI study introduces an 'intelligence per watt' metric to gauge how efficiently local AI models convert energy into usable capability.... og:url: https://daily.dev/posts/stanford-and-together-ai-study-local-ai-efficiency-improved-5-3-since-2023-but-hard-reasoning-tas-0cuniozjt og:image: https://api.daily.dev/og/posts/0CunioZjt.png og:image:alt: Stanford and Together AI study: local AI efficiency improved 5.3× since 2023, but hard reasoning tasks remain a wall og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Stanford and Together AI study: local AI efficiency improved 5.3× since 2023, but hard reasoning tasks remain a wall **[Collections](https://daily.dev/sources/collections)** · 2 min read · 1 upvotes · 0 comments ## Summary A Stanford and Together AI study introduces an 'intelligence per watt' metric to gauge how efficiently local AI models convert energy into usable capability. From 2023 to 2025, local models' intelligence-per-watt improved 5.3x, and the share of queries they can handle jumped from 23.2% to 71.3%. Hybrid routing between local and cloud models cut energy, compute, and cost by 60-80% versus always using cloud. An iPhone 16 Pro hit about 7x higher intelligence-per-watt than workstation GPUs on the same model and precision. A pool of 20+ local models, each routed to its best-fit query, beat three frontier cloud models on 3 of 4 benchmarks. Dropping precision from FP16 to FP4 cut inference energy 3-3.5x for roughly 2.5 accuracy points per step, and one larger FP4 model even beat a smaller FP16 model. The persistent weak spot: about 95% of the hardest reasoning problems remain unsolved by local models, meaning cloud models still matter for the toughest tasks. ## Content A new paper from Stanford and Together AI introduces a metric called "intelligence per watt" to measure how efficiently local AI models use energy relative to their actual capability. The headline numbers are striking. From 2023 to 2025, local AI's intelligence-per-watt improved 5.3×, and the share of queries that local models can handle jumped from 23.2% to 71.3%. That's a lot of ground covered in two years. A few specific findings worth noting: **Hybrid routing cuts costs dramatically.** Routing queries between local and cloud models based on difficulty reduced energy, compute, and cost by 60–80% compared to just sending everything to a cloud baseline. **Mobile hardware punches above its weight.** An iPhone 16 Pro achieved roughly 7× higher intelligence-per-watt than workstation GPUs running the same model at the same precision. For lightweight queries, mobile AI is surprisingly efficient. **Model diversity beats frontier models on most benchmarks.** A pool of 20+ local models outperformed three frontier cloud models on 3 of 4 benchmarks, when each query was routed to whichever local model handled it best. The diversity of the pool mattered more than any single model's raw capability. **Lower precision saves a lot of energy for a small accuracy cost.** Going from FP16 to FP4 cut inference energy by 3–3.5×, at a cost of roughly 2.5 percentage points of accuracy per precision step. In at least one test, a larger FP4 model beat a smaller FP16 model outright. The one area where local models still fall flat: hard reasoning. About 95% of the hardest problems in the benchmark remained unsolved by local models. Easier and medium-difficulty tasks improved rapidly, but the ceiling on complex reasoning hasn't moved much. That's where cloud models still earn their keep. The broader implication is that "always use the biggest cloud model" is increasingly hard to justify for most queries. Smarter routing, lower precision, and a diverse local model pool can get you most of the way there at a fraction of the energy cost. ## Questions this post answers ### How much has local AI model efficiency improved compared to 2023? Local AI models' intelligence-per-watt efficiency improved 5.3x between 2023 and 2025, according to a joint Stanford and Together AI study. Over the same period, the share of queries that local models could handle jumped from 23.2% to 71.3%, reflecting rapid gains on easy and medium-difficulty tasks even though the hardest reasoning problems remain largely unsolved. _Track efficiency research like this on daily.dev to inform decisions on local versus cloud AI deployment._ ### How much energy does dropping from FP16 to FP4 precision save for local model inference, and what's the accuracy tradeoff? Going from FP16 to FP4 precision cuts inference energy by roughly 3 to 3.5 times, at a cost of about 2.5 percentage points of accuracy per precision step. In at least one measured case, a larger FP4 model actually outperformed a smaller FP16 model, suggesting quantization tradeoffs aren't always a straightforward loss. _Developers weighing quantization tradeoffs can follow findings like these on daily.dev._ ### Can a pool of local AI models outperform frontier cloud models like GPT-4 class systems? Yes, a pool of more than 20 local models, each query routed to whichever model handled it best, outperformed three frontier cloud models on 3 out of 4 benchmarks. The diversity of the model pool mattered more than any single model's raw capability, though the hardest reasoning problems still favored cloud models, with about 95% of the toughest tasks unsolved locally. _Anyone deciding between cloud and local model strategies can follow this kind of routing research on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 3 discussions and 18 comments across x (as of 2026-09-11). **TL;DR:** Replies mostly welcome the efficiency gains and see smart routing between local and cloud models as the real breakthrough, while some question the study's methodology and originality. **Sentiment:** 40% positive · 45% mixed · 15% skeptical **The case for** - Hybrid routing cutting energy/cost 60-80% is seen as a bigger deal than most new model announcements. - An iPhone beating workstation GPUs on intelligence-per-watt is viewed as a strong signal that local-first inference is winning. - Keeping queries local is valued for compliance/privacy reasons, avoiding cross-border data transfer reviews. **The pushback** - Some question whether the 3-of-4 benchmark win is realistic if the best local model was picked after scoring all of them, calling it a ceiling rather than something runnable in practice. - One reply disputes the study's originality, claiming similar metrics were discussed elsewhere well before this study and accusing Stanford of not crediting prior work. - One reply frames the routing savings cynically as mainly a financial/capex win rather than a privacy or intelligence story. **By community** - x (positive): Replies generally praise the efficiency gains and routing insights, with only scattered skepticism about methodology and originality. **Hottest debate:** Whether the 3-of-4 benchmark win from routing reflects a realistic, deployable system or just a best-case ceiling from post-hoc model selection. **Open questions** - How exactly was the best-fit local model selected for each query, and could that selection process be replicated in a real deployment? **Highlights** > @rohanpaul_ai how did they pick which local model gets each query? if that choice was made after scoring all of them, the 3-of-4 result is a ceiling rather than something you could run > — [unpaidneuron on x](https://x.com/unpaidneuron/status/2098306850177650939) > @rohanpaul_ai A 60-80% cut in energy and cost from hybrid local-cloud routing is a bigger deal than most new model announcements this week. Routing decisions are quietly becoming as important as the models themselves. > — [S\_Fadaeimanesh on x](https://x.com/S_Fadaeimanesh/status/2098307851953267037) > @rohanpaul_ai The routing layer is the part regulated teams will care about. If a healthcare query gets pushed to a local FP4 model to save power and it gets something wrong, you need a record of why it was routed there. Accuracy alone will not satisfy an auditor. > — [AnandNSharma on x](https://x.com/AnandNSharma/status/2098314315988365403) > @rohanpaul_ai Cool, but I doubt its original at all! @BrianRoemmele been talking about these same measures for at least a year or more. AND Stanford is the institution well known to steal stuff from other less recognized institutions and scientist. > — [DemiClawOne on x](https://x.com/DemiClawOne/status/2098414242676220202) > @rohanpaul_ai 60-80% savings but the real story is capex avoidance. if routing logic can predict which queries need cloud-scale reasoning vs local, you're basically building a traffic cop for GPU spend. that's the product, not the benchmark. > — [AllianceDouble on x · 1 points](https://x.com/AllianceDouble/status/2098302537749430311) **Source threads** - [x](https://x.com/rohanpaul_ai/status/2098555026776195134) · 0 points · 0 comments - [x](https://x.com/omarsar0/status/2098323417229631703) · 0 points · 0 comments - [x](https://x.com/rohanpaul_ai/status/2098301299032990059) · 0 points · 18 comments ## Similar posts on daily.dev - [Import AI 435: 100k training runs; AI systems absorb human power; intelligence per watt](https://daily.dev/posts/import-ai-435-100k-training-runs-ai-systems-absorb-human-power-intelligence-per-watt-8po52yc6c) · Import AI · 0 upvotes · 0 comments - [Local AI is more accessible than ever, but with one major GPU-sized caveat](https://daily.dev/posts/local-ai-is-more-accessible-than-ever-but-with-one-major-gpu-sized-caveat-ozobd3gds) · XDA Developers · 0 upvotes · 0 comments - [No, local models will not win](https://daily.dev/posts/no-local-models-will-not-win-t3ebfknyr) · sean goedecke · 2 upvotes · 2 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#local-ai](https://daily.dev/tags/local-ai) [View this post on daily.dev](https://daily.dev/posts/stanford-and-together-ai-study-local-ai-efficiency-improved-5-3-since-2023-but-hard-reasoning-tas-0cuniozjt) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Stanford and Together AI study: local AI efficiency improved 5.3× since 2023, but hard reasoning tasks remain a wall","url":"https://daily.dev/posts/stanford-and-together-ai-study-local-ai-efficiency-improved-5-3-since-2023-but-hard-reasoning-tas-0cuniozjt","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/stanford-and-together-ai-study-local-ai-efficiency-improved-5-3-since-2023-but-hard-reasoning-tas-0cuniozjt"},"datePublished":"2026-09-11T23:31:46.183Z","dateModified":"2026-09-11T23:32:37.783Z","description":"A Stanford and Together AI study introduces an 'intelligence per watt' metric to gauge how efficiently local AI models convert energy into usable capability....","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/stanford-and-together-ai-study-local-ai-efficiency-improved-5-3-since-2023-but-hard-reasoning-tas-0cuniozjt","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-science,local-ai","timeRequired":"PT2M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"Stanford and Together AI study: local AI efficiency improved 5.3× since 2023, but hard reasoning tasks remain a wall"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/stanford-and-together-ai-study-local-ai-efficiency-improved-5-3-since-2023-but-hard-reasoning-tas-0cuniozjt#faq","mainEntity":[{"@type":"Question","name":"How much has local AI model efficiency improved compared to 2023?","acceptedAnswer":{"@type":"Answer","text":"Local AI models' intelligence-per-watt efficiency improved 5.3x between 2023 and 2025, according to a joint Stanford and Together AI study. Over the same period, the share of queries that local models could handle jumped from 23.2% to 71.3%, reflecting rapid gains on easy and medium-difficulty tasks even though the hardest reasoning problems remain largely unsolved. Track efficiency research like this on daily.dev to inform decisions on local versus cloud AI deployment."}},{"@type":"Question","name":"How much energy does dropping from FP16 to FP4 precision save for local model inference, and what's the accuracy tradeoff?","acceptedAnswer":{"@type":"Answer","text":"Going from FP16 to FP4 precision cuts inference energy by roughly 3 to 3.5 times, at a cost of about 2.5 percentage points of accuracy per precision step. In at least one measured case, a larger FP4 model actually outperformed a smaller FP16 model, suggesting quantization tradeoffs aren't always a straightforward loss. Developers weighing quantization tradeoffs can follow findings like these on daily.dev."}},{"@type":"Question","name":"Can a pool of local AI models outperform frontier cloud models like GPT-4 class systems?","acceptedAnswer":{"@type":"Answer","text":"Yes, a pool of more than 20 local models, each query routed to whichever model handled it best, outperformed three frontier cloud models on 3 out of 4 benchmarks. The diversity of the model pool mattered more than any single model's raw capability, though the hardest reasoning problems still favored cloud models, with about 95% of the toughest tasks unsolved locally. Anyone deciding between cloud and local model strategies can follow this kind of routing research on daily.dev."}}]} ```