--- title: A 3.66B model just embarrassed the 'small model' leaderboard description: A new 3.66B-parameter model from webAI, TwIL-LM3-Pro, built specifically for formal logic tasks like checking deductions and critiquing proofs, is drawing... canonical: https://daily.dev/posts/a-3-66b-model-just-embarrassed-the-small-model-leaderboard-asu5n1qcm twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: A 3.66B model just embarrassed the 'small model' leaderboard | daily.dev og:description: A new 3.66B-parameter model from webAI, TwIL-LM3-Pro, built specifically for formal logic tasks like checking deductions and critiquing proofs, is drawing... og:url: https://daily.dev/posts/a-3-66b-model-just-embarrassed-the-small-model-leaderboard-asu5n1qcm og:image: https://api.daily.dev/og/posts/asu5n1qcm.png og:image:alt: A 3.66B model just embarrassed the 'small model' leaderboard og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # A 3.66B model just embarrassed the 'small model' leaderboard **[Trends](https://daily.dev/sources/trends)** · 2 min read · 2 upvotes · 0 comments ## Summary A new 3.66B-parameter model from webAI, TwIL-LM3-Pro, built specifically for formal logic tasks like checking deductions and critiquing proofs, is drawing attention after beating larger small-model competitors on benchmarks. webAI claims it beats VibeThinker-3B by roughly 35%, Qwen3.5-4B by 24%, and LFM2.5-8B-A1B by 47% on formal-logic scoring, and scores 94.3% on GSM8K versus 97.7% for the 120B-parameter gpt-oss model. It runs as a 2.09GiB Q4 quantized build via llama.cpp on CPU or local GPU. All figures come from webAI's own evaluations rather than independent replication, and the model is narrowly specialized for formal logic rather than general-purpose use. ## Content The framing on X is already a geopolitics scoreboard. @rohanpaul_ai: "China's best small reasoning model (VibeThinker-3B) just got beaten by a 3.6B model from Austin." The model is webAI's TwIL-LM3-Pro, and the claim is that it leads every small model webAI compared on formal logic. ## The numbers being passed around webAI's own comparisons, as relayed by @rohanpaul_ai, put it roughly 35% above Weibo's VibeThinker-3B, 24% above Qwen3.5-4B and 47% above Liquid AI's LFM2.5-8B-A1B. It also lifts IBM Granite's formal-logic score by 28%. @omarsar0 and @svpino both cite 95.4 on BIG-Bench Hard's logic subset. @omarsar0 contrasts that with Qwen3-8B at 63.7, a model more than twice its size. @Hesamation goes for the size-efficiency angle. On webAI's GSM8K evaluation, TwIL-LM3-Pro scores 94.3% against gpt-oss-120b's 97.7%, with "x33 FEWER PARAMS." His takeaway: "you don't need a massive model for every little problem." All of these figures come from webAI's own evals. The excitement is running well ahead of any independent replication. ## Why developers are into it The practical pitch is the draw. The recommended Q4 build is a 2.09GiB file that runs through llama.cpp on CPU or a local GPU, so private data can stay on the device. @svpino says it's small enough to run on a laptop. Then there's the recipe. @omarsar0 says he likes the post-training approach: fine-tune on formal logic, merge the weights back toward the base model, then run RL against a programmatic verifier. His read is that logic scores go up while general reasoning holds steady. That merge step is the part that sounds most reusable. @omarsar0 also frames small models as an opening at the harness layer, where a cheap local reasoner can be slotted into a larger system. And he's glad to see more of this released as open source. @svpino's lesson is broader: specialized post-training works, and "you can get a ton of small models when you focus on specific tasks." ## The catch This is a specialist. It was built for checking deductions and critiquing proofs, and the wins are on logic and math word problems. The gpt-oss-120b comparison also shows it still trails the big model on GSM8K, just not by much. "Beats China's best" and "close to a 120B" are both true only on the benchmarks webAI picked. Still, a 2GB file doing this on a CPU is the part that sticks. ## Questions this post answers ### How does webAI's TwIL-LM3-Pro 3.66B model compare to gpt-oss 120B on GSM8K? TwIL-LM3-Pro scores 94.3% on webAI's GSM8K evaluation compared to 97.7% for the 120B-parameter gpt-oss model, a 3.4 percentage point gap despite being roughly 33 times smaller. It also reportedly beats VibeThinker-3B, Qwen3.5-4B, and LFM2.5-8B-A1B on formal-logic scoring by double-digit margins, though all figures come from webAI's own benchmarking rather than independent replication. _Developers comparing small versus large model trade-offs can follow new benchmark claims like these on daily.dev._ ### Can I run a formal logic reasoning model locally on a laptop without a GPU? Yes, the recommended Q4 quantized build of TwIL-LM3-Pro is a 2.09GiB file that runs through llama.cpp on CPU or a local GPU, keeping private data on-device. It scores 95.4% on BIG-Bench Hard's logic subset, suggesting specialized small models can handle formal logic tasks on commodity hardware instead of calling a large hosted API. _Those weighing local inference versus cloud APIs can track small-model developments like this on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 9 comments across x (as of 2026-10-01). **TL;DR:** People are intrigued by a small model hitting a high BIG-Bench Hard score, but several push back on whether the benchmark numbers are trustworthy or representative, and others question how much credit goes to the model versus the harness around it. **Sentiment:** 30% positive · 35% mixed · 35% skeptical **The case for** - Stronger reasoning in small models could make local, on-device AI more practical. - Running near-95 BBH performance on a laptop-scale model is seen as a meaningful step for local harness work. **The pushback** - The gap between the model's score and larger models on BBH seems suspiciously large, raising contamination concerns until independently verified blind. - Benchmark numbers can be inflated by settings like reasoning-effort, so the reported score may not reflect default real-world performance. - On a held-out track another model reportedly scores higher, suggesting the gains may be narrowly in-domain rather than general. - BIG-Bench Hard is synthetic, leaving doubts about performance on messier real-world tasks like debugging or planning with incomplete context. **By community** - x (mixed): Interest in small-model reasoning gains is tempered by pointed skepticism about benchmark validity, settings used, and generalization. **Hottest debate:** Whether the reported BIG-Bench Hard score reflects genuine reasoning gains or is inflated/contaminated and narrowly in-domain. **Open questions** - How much of the reported score survives contact with real toolchains outside the benchmark? - What reasoning-effort setting was used to produce the 95.4 score? - Is BBH included in the held-out evaluation track, and how does performance there compare? - How does the model perform on messier real-world tasks like debugging code or planning with incomplete context? **Highlights** > @omarsar0 @thewebAI 95.4 vs 63.7 on BBH from a model under half the size? that gap is too clean, smells like benchmark contamination until someone runs it blind 🥴 > — [Aynazendaya on x](https://x.com/Aynazendaya/status/2105743328818213235) > @omarsar0 @thewebAI Worth checking what reasoning-effort setting produced that 95.4 - I onboarded a cheap long-context model that scored weak by default and only hit its marketing numbers at high effort. For small models that knob matters as much as the benchmark. > — [MaximTitarenko on x](https://x.com/MaximTitarenko/status/2105731611992174960) > @omarsar0 @thewebAI On the held-out track in your chart Qwen3-8B is ahead though, 0.759 to 0.743 on the all-14 macro. Reads like the logic training lands in-domain. Is BBH inside Track B? > — [AIQuanting on x](https://x.com/AIQuanting/status/2105732650053108159) > @omarsar0 @thewebAI 95.4 is impressive but BIG-Bench Hard is still synthetic. what does the reasoning look like on something messier, like debugging real code or planning with incomplete context? > — [jatingargiitk on x](https://x.com/jatingargiitk/status/2105736634595696775) > @omarsar0 @thewebAI the harness layer thesis is the thing. when reasoning runs local at 95.4 on big-bench hard, the model becomes plumbing and the orchestration becomes the product. the open question for me: how much of that score survives contact with real toolchains — or are we about to find out > — [Zhen4good on x](https://x.com/Zhen4good/status/2105747537261138052) **Source threads** - [x](https://x.com/omarsar0/status/2105730500811719056) · 0 points · 9 comments ## Similar posts on daily.dev - [Claude, ChatGPT, and Gemini get all the hype, but the most interesting AI models are coming from elsewhere](https://daily.dev/posts/claude-chatgpt-and-gemini-get-all-the-hype-but-the-most-interesting-ai-models-are-coming-from-els-glbavr35d) · XDA Developers · 0 upvotes · 0 comments - [Qwen3.6–35B-A3B: The Most Practical Open-Source AI Model Yet?](https://daily.dev/posts/qwen3-6-35b-a3b-the-most-practical-open-source-ai-model-yet--eihfhk9n5) · Faun · 77 upvotes · 6 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#llama-cpp](https://daily.dev/tags/llama-cpp) [View this post on daily.dev](https://daily.dev/posts/a-3-66b-model-just-embarrassed-the-small-model-leaderboard-asu5n1qcm) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"A 3.66B model just embarrassed the 'small model' leaderboard","url":"https://daily.dev/posts/a-3-66b-model-just-embarrassed-the-small-model-leaderboard-asu5n1qcm","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/a-3-66b-model-just-embarrassed-the-small-model-leaderboard-asu5n1qcm"},"datePublished":"2026-10-01T00:35:52.750Z","dateModified":"2026-10-01T21:45:18.651Z","description":"A new 3.66B-parameter model from webAI, TwIL-LM3-Pro, built specifically for formal logic tasks like checking deductions and critiquing proofs, is drawing...","image":"https://pbs.twimg.com/media/HTgVgU_WIAA0Bbq.jpg","thumbnailUrl":"https://pbs.twimg.com/media/HTgVgU_WIAA0Bbq.jpg","isAccessibleForFree":true,"articleSection":"Trends","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Trends","logo":"https://media.daily.dev/image/upload/s--ZfSp3asX--/f_auto,q_auto/v1780996004/logos/trends?_a=BAMAMiWQ0","url":"https://daily.dev/sources/trends"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/a-3-66b-model-just-embarrassed-the-small-model-leaderboard-asu5n1qcm","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,llama-cpp","timeRequired":"PT2M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Trends","item":"https://daily.dev/sources/trends"},{"@type":"ListItem","position":3,"name":"A 3.66B model just embarrassed the 'small model' leaderboard"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/a-3-66b-model-just-embarrassed-the-small-model-leaderboard-asu5n1qcm#faq","mainEntity":[{"@type":"Question","name":"How does webAI's TwIL-LM3-Pro 3.66B model compare to gpt-oss 120B on GSM8K?","acceptedAnswer":{"@type":"Answer","text":"TwIL-LM3-Pro scores 94.3% on webAI's GSM8K evaluation compared to 97.7% for the 120B-parameter gpt-oss model, a 3.4 percentage point gap despite being roughly 33 times smaller. It also reportedly beats VibeThinker-3B, Qwen3.5-4B, and LFM2.5-8B-A1B on formal-logic scoring by double-digit margins, though all figures come from webAI's own benchmarking rather than independent replication. Developers comparing small versus large model trade-offs can follow new benchmark claims like these on daily.dev."}},{"@type":"Question","name":"Can I run a formal logic reasoning model locally on a laptop without a GPU?","acceptedAnswer":{"@type":"Answer","text":"Yes, the recommended Q4 quantized build of TwIL-LM3-Pro is a 2.09GiB file that runs through llama.cpp on CPU or a local GPU, keeping private data on-device. It scores 95.4% on BIG-Bench Hard's logic subset, suggesting specialized small models can handle formal logic tasks on commodity hardware instead of calling a large hosted API. Those weighing local inference versus cloud APIs can track small-model developments like this on daily.dev."}}]} ```