--- title: Evaluating a decision model: Nimble 9B and its base... description: An independent benchmark compares Nimble 9B, a 'decision model' released via Ollama, against its own base model Qwen3.5-9B on a 900-sample routing task using... canonical: https://daily.dev/posts/evaluating-a-decision-model-nimble-9b-and-its-base-model-qwen3-5-9b-orw9bh3z5 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Evaluating a decision model: Nimble 9B and its base model Qwen3.5–9B | daily.dev og:description: An independent benchmark compares Nimble 9B, a 'decision model' released via Ollama, against its own base model Qwen3.5-9B on a 900-sample routing task using... og:url: https://daily.dev/posts/evaluating-a-decision-model-nimble-9b-and-its-base-model-qwen3-5-9b-orw9bh3z5 og:image: https://api.daily.dev/og/posts/oRw9bH3z5.png og:image:alt: Evaluating a decision model: Nimble 9B and its base model Qwen3.5–9B og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Evaluating a decision model: Nimble 9B and its base model Qwen3.5–9B **[Dev Genius](https://daily.dev/sources/devgenius)** · 24 min read · 0 upvotes · 0 comments ## Summary An independent benchmark compares Nimble 9B, a 'decision model' released via Ollama, against its own base model Qwen3.5-9B on a 900-sample routing task using real CFPB consumer complaints. Accuracy and speed turn out nearly identical between the two models (77.1% vs 76.3%, both around 2 seconds per request), contradicting vendor claims of large accuracy gains and sub-100ms latency (which only appeared on exact cache hits). The one real advantage found was better-calibrated confidence scores: using probability thresholds, Nimble could safely automate 43% of complaints at 95% accuracy versus only 16% for the base model. Enabling reasoning on the base model added about 3 points of accuracy but took 38x longer. The piece concludes a decision model is best used as a first-pass confidence gate in an automate-or-escalate pipeline, not as a universal speed or accuracy upgrade. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Is Nimble 9B actually faster than its base model Qwen3.5-9B for routing tasks on Ollama? No, Nimble is slightly slower in practice. On a 900-complaint routing benchmark, Nimble's median response time was 2.25 seconds versus 2.07 seconds for Qwen3.5-9B with reasoning disabled. Nimble's shorter fixed output cost (0.12s vs 0.69s for Qwen) is offset by its longer input format, which adds about 170 extra tokens per request compared to Qwen's prompt. _Developers benchmarking decision models against chat models for routing pipelines can compare real numbers like these on daily.dev._ ### Does Nimble 9B really achieve 91ms response times as Ollama's announcement claims? Only when sending an identical repeated request, which hits an internal cache and returns in about 0.10 seconds. For any new or even slightly modified complaint, measured response time on an M1 Pro MacBook Pro was about 2.25 seconds median, roughly 25 times slower than the published 91ms figure (measured on a faster M5 Max chip). A short new request still took around 1 second. _Anyone sizing latency claims for AI inference tools before committing to an architecture can track findings like this on daily.dev._ ### Does enabling reasoning mode on Qwen3.5-9B improve accuracy enough to justify the extra time cost? Reasoning raised accuracy by about 3 points, from 76.3% to 79.6%, on a 900-item complaint routing task, but took 38 times longer: 20 hours 35 minutes versus 33 minutes with reasoning off. The reasoning model wrote a median of 862 tokens per answer versus about 10 without reasoning, and longer reasoning chains correlated with more wrong answers, suggesting reasoning length can signal ambiguous input. _Teams weighing reasoning-mode latency against accuracy gains for LLM routing can follow comparisons like this on daily.dev._ ## Similar posts on daily.dev - [Choosing the Right Model for Your Inference Use Case](https://daily.dev/posts/choosing-the-right-model-for-your-inference-use-case-7usyrqlyu) · DigitalOcean Community · 2 upvotes · 0 comments - [The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents](https://daily.dev/posts/the-agentic-analytics-benchmark-measuring-model-accuracy-and-efficiency-in-analytical-agents-81a3conc5) · ClickHouse · 0 upvotes · 0 comments - [The Agent Said It Was Done. The Database Disagreed.](https://daily.dev/posts/the-agent-said-it-was-done-the-database-disagreed--th0ihs2vh) · Hugging Face · 2 upvotes · 0 comments - [AWS Open-Sourced a Decision Model as an Agent](https://daily.dev/posts/aws-open-sourced-a-decision-model-as-an-agent-wdbnayxnz) · Medium · 1 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#local-ai](https://daily.dev/tags/local-ai), [#ollama](https://daily.dev/tags/ollama), [#qwen](https://daily.dev/tags/qwen) [View this post on daily.dev](https://daily.dev/posts/evaluating-a-decision-model-nimble-9b-and-its-base-model-qwen3-5-9b-orw9bh3z5) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Evaluating a decision model: Nimble 9B and its base model Qwen3.5–9B","url":"https://daily.dev/posts/evaluating-a-decision-model-nimble-9b-and-its-base-model-qwen3-5-9b-orw9bh3z5","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/evaluating-a-decision-model-nimble-9b-and-its-base-model-qwen3-5-9b-orw9bh3z5"},"datePublished":"2026-10-12T18:40:57.451Z","dateModified":"2026-10-12T19:07:50.590Z","description":"An independent benchmark compares Nimble 9B, a 'decision model' released via Ollama, against its own base model Qwen3.5-9B on a 900-sample routing task using...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/372cb790e3f285ad247c446e3596d5af?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/372cb790e3f285ad247c446e3596d5af?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Dev Genius","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Dev Genius","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/a79144d7517043e7803ed562e45a99df","url":"https://daily.dev/sources/devgenius"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/evaluating-a-decision-model-nimble-9b-and-its-base-model-qwen3-5-9b-orw9bh3z5","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,local-ai,ollama,qwen","timeRequired":"PT24M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Dev Genius","item":"https://daily.dev/sources/devgenius"},{"@type":"ListItem","position":3,"name":"Evaluating a decision model: Nimble 9B and its base model Qwen3.5–9B"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/evaluating-a-decision-model-nimble-9b-and-its-base-model-qwen3-5-9b-orw9bh3z5#faq","mainEntity":[{"@type":"Question","name":"Is Nimble 9B actually faster than its base model Qwen3.5-9B for routing tasks on Ollama?","acceptedAnswer":{"@type":"Answer","text":"No, Nimble is slightly slower in practice. On a 900-complaint routing benchmark, Nimble's median response time was 2.25 seconds versus 2.07 seconds for Qwen3.5-9B with reasoning disabled. Nimble's shorter fixed output cost (0.12s vs 0.69s for Qwen) is offset by its longer input format, which adds about 170 extra tokens per request compared to Qwen's prompt. Developers benchmarking decision models against chat models for routing pipelines can compare real numbers like these on daily.dev."}},{"@type":"Question","name":"Does Nimble 9B really achieve 91ms response times as Ollama's announcement claims?","acceptedAnswer":{"@type":"Answer","text":"Only when sending an identical repeated request, which hits an internal cache and returns in about 0.10 seconds. For any new or even slightly modified complaint, measured response time on an M1 Pro MacBook Pro was about 2.25 seconds median, roughly 25 times slower than the published 91ms figure (measured on a faster M5 Max chip). A short new request still took around 1 second. Anyone sizing latency claims for AI inference tools before committing to an architecture can track findings like this on daily.dev."}},{"@type":"Question","name":"Does enabling reasoning mode on Qwen3.5-9B improve accuracy enough to justify the extra time cost?","acceptedAnswer":{"@type":"Answer","text":"Reasoning raised accuracy by about 3 points, from 76.3% to 79.6%, on a 900-item complaint routing task, but took 38 times longer: 20 hours 35 minutes versus 33 minutes with reasoning off. The reasoning model wrote a median of 862 tokens per answer versus about 10 without reasoning, and longer reasoning chains correlated with more wrong answers, suggesting reasoning length can signal ambiguous input. Teams weighing reasoning-mode latency against accuracy gains for LLM routing can follow comparisons like this on daily.dev."}}]} ```