--- title: NVIDIA releases Nemotron 3.5 Lightning and NeMo... description: NVIDIA has released two tools targeting AI agent workloads: Nemotron 3.5 Lightning, a 30B mixture-of-experts model with only 3B active parameters at inference,... canonical: https://daily.dev/posts/nvidia-releases-nemotron-3-5-lightning-and-nemo-switchyard-for-ai-agent-workloads-8kkbicyjr twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: NVIDIA releases Nemotron 3.5 Lightning and NeMo Switchyard for AI agent workloads | daily.dev og:description: NVIDIA has released two tools targeting AI agent workloads: Nemotron 3.5 Lightning, a 30B mixture-of-experts model with only 3B active parameters at inference,... og:url: https://daily.dev/posts/nvidia-releases-nemotron-3-5-lightning-and-nemo-switchyard-for-ai-agent-workloads-8kkbicyjr og:image: https://api.daily.dev/og/posts/8KkbIcyjr.png og:image:alt: NVIDIA releases Nemotron 3.5 Lightning and NeMo Switchyard for AI agent workloads og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # NVIDIA releases Nemotron 3.5 Lightning and NeMo Switchyard for AI agent workloads **[Collections](https://daily.dev/sources/collections)** · 5 min read · 3 upvotes · 0 comments ## Summary NVIDIA has released two tools targeting AI agent workloads: Nemotron 3.5 Lightning, a 30B mixture-of-experts model with only 3B active parameters at inference, and NeMo Switchyard, an open-source routing library. Nemotron 3.5 Lightning uses speculative decoding, NVFP4 quantization, and dual draft models to achieve 86% accuracy on PinchBench while completing 10,000 tasks 30% faster than Qwen3 35B. It runs on consumer hardware and deploys via vLLM, Ollama, and llama.cpp under the OpenMDW-1.1 license. NeMo Switchyard routes each agent step to the most cost-effective model based on accuracy, latency, and cost signals. Real-world results include a 74% cost reduction in LangChain testing with only a ~6-point accuracy drop, and Cognition's Devin Desktop achieving 50.6% accuracy at $3.11 mean cost — within 2.8 points of frontier accuracy at ~28% lower cost. ## Content NVIDIA just shipped two things that, taken together, say more about where AI infrastructure is heading than either one does alone: a small, fast open model called Nemotron 3.5 Lightning, and a routing library called NeMo Switchyard that decides when to actually use it. I think the pairing matters more than the individual releases, so let me walk through both and then get into why I think this is a bigger deal than "NVIDIA released a model and a router." ## The model: Nemotron 3.5 Lightning Lightning is a 30B-parameter mixture-of-experts model with only 3B active per forward pass, built on a hybrid Mamba-2 + MoE + attention architecture. It's not trying to be a frontier reasoning model. It's built for the boring, high-volume, always-on parts of agent work — tool calls, retrieval, summarization, classification, result validation — the stuff that happens hundreds of times per task while a bigger model handles the planning. The numbers NVIDIA is putting forward: - Up to 4x higher throughput (~410 tokens/sec) versus comparable models - 30-35% faster task completion than Qwen3.6 35B at similar accuracy - 86% accuracy on PinchBench across 10,000 tasks - 52.80 on SWE-bench Verified, 75.57 on GPQA Diamond - Up to 1M token context via DFlash speculative decoding A lot of that speed comes from stacking multiple tricks: a built-in multi-token predictor, two speculative decoding drafters (DSpark and DFlash, the latter diffusion-based), and NVFP4 quantization. NVIDIA also published a companion piece on how they built the NVFP4 checkpoint using quantization-aware distillation (QAD) — basically post-training quantization first, then distilling the full-precision teacher into the quantized student using a KL divergence loss on logits. That process took the model from 66GB down to 22GB while recovering most of the accuracy that aggressive quantization would normally cost, particularly on agentic and reasoning benchmarks. The model ships fully open — weights, training data, and recipes — under the OpenMDW-1.1 license, pre-trained on 20T+ tokens. That openness is showing up fast in fine-tuning results people are actually reporting: CrowdStrike says they hit Nemotron 3 Ultra-level accuracy at a fifth of the cost after fine-tuning; CodeRabbit and Base10 trained it for one epoch in under three hours for about $100. Unsloth got a 2-bit quantized version running tool calls nonstop for 10 minutes on 22GB of VRAM, citing over 80 websites along the way — which, for a 30B model, is a genuinely impressive party trick.< br> Deployment support arrived fast too. vLLM shipped day-0 support with continuous batching, prefix caching, and all three speculative decoding strategies, plus some real engineering underneath: a Hopper-optimized backend replacing Marlin for roughly 20% more throughput, a quantized draft head, and a rewritten SSM path for the Mamba2 layers. It also runs via Ollama, llama.cpp, LM Studio, TensorRT-LLM, SGLang, and is available on Hugging Face, SageMaker JumpStart, and even as an Ubuntu snap through Canonical. Hardware support spans a single H100 or DGX Spark up to 8×H100 and full data center clusters, plus RTX 5090s and Jetson for edge deployment. Worth flagging: this thing is explicitly not designed to be your front-line orchestrator. It's weak on prompt injection resistance and isn't meant for high-level reasoning. It's the execution layer. Which brings us to the part I find more interesting. ## The router: NeMo Switchyard Switchyard is an open-source, Rust-based routing library that sits between an agent and a pool of available models, deciding which model handles each individual step. Not each request — each *step*. That distinction turns out to matter a lot. It offers two families of routers: - **Tuning-free**: LLM classifier, stage router, escalation router - **Tunable**: prefill router It runs on CPU, needs no GPU RAM, translates between OpenAI, Anthropic, and Responses API formats, and integrates with LiteLLM, Kong, OpenRouter, and LangChain. NVIDIA claims 50% faster responses and 25% better token efficiency out of the box. The early partner numbers are the part that actually convinced me this isn't just a nice-to-have: - **LangChain**: 74% cost reduction, sending only 7% of calls to a frontier model - **Ramp**: 58% lower costs and 33% faster runtime while matching frontier performance on SWE-Bench - **Cognition (Devin Desktop)**: 50.6% accuracy at $3.11 mean cost — within 2.8 points of frontier accuracy at roughly 28% lower cost A more detailed benchmark run — 145 multi-step agentic tasks averaging 6.3 model calls each — found that only 7% of turns actually needed Claude Opus 4.8. Lightning handled the rest. Total cost dropped 74%, at a cost of about 6 accuracy points. There's a useful break-even formula buried in that writeup too: minimum offload = judge cost / (expensive cost − cheap cost). If your cheap and expensive models are priced close together, routing won't pay off unless the cheap model is self-hosted. Also worth noting: the judge model itself ate up 21.2% of routed spend, which the authors flag as the highest-leverage place left to optimize. And routing isn't a good fit everywhere — it's not recommended for latency-sensitive or short single-turn workloads. ## Why I think this is the real story Here's the thing that keeps nagging at me: everyone's calling Switchyard an ## Questions this post answers ### How much faster is Nemotron 3.5 Lightning compared to Qwen3 35B on agent benchmarks? Nemotron 3.5 Lightning completes 10,000 tasks 30% faster than Qwen3 35B at comparable accuracy, hitting 86% on PinchBench. NVIDIA also claims up to 4x faster token generation versus similar open models. The model achieves this using speculative decoding with multi-token prediction, NVFP4 quantization, and two draft models (DSpark and DFlash), while keeping only 3B of its 30B parameters active at inference. _Teams benchmarking open models for agent workloads track comparisons like this on daily.dev._ ### How much can NeMo Switchyard reduce AI agent inference costs in practice? NeMo Switchyard's escalation router cut costs by 74% in LangChain testing with only a ~6-point accuracy drop, achieved by routing just 7% of calls to the frontier model. In Cognition's Devin Desktop deployment, staged routing reached 50.6% accuracy at $3.11 mean cost — within 2.8 percentage points of frontier accuracy at roughly 28% lower cost. NVIDIA also reports benchmark completion costs around one-third of using Opus 4.8 alone. _Developers optimizing agent pipeline spend find the latest cost-vs-accuracy tradeoffs on daily.dev._ ### What hardware can run Nemotron 3.5 Lightning locally? Nemotron 3.5 Lightning runs on consumer hardware including the GeForce RTX 5090, DGX Spark, Jetson, and OEM GB10 systems, and scales to data centers and cloud. Deployment is supported through vLLM, Ollama, llama.cpp, LM Studio, and Unsloth, with NVFP4 and GGUF formats available. Weights, training data, and recipes are fully open under the OpenMDW-1.1 license. _Developers self-hosting open models for agent workloads follow hardware and deployment news on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 2 discussions and 92 comments across hackernews (as of 2026-08-17). **TL;DR:** NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI: The community is broadly interested in the trend toward smaller, efficient models that Nemotron 3.5 Lightning represents, but discussion quickly diverges into debates about whether small models are the future or just a stopgap, and there is notable skepticism about NeMo Switchyard's prompt-caching compatibility and its 'experimental, not for production' status. nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 · Hugging Face: The community sees Nemotron-3.5 Lightning as a genuinely interesting open-source model with a compelling hybrid architecture and fully open training pipeline, though most agree it trails Qwen 3.6 35B-A3B on benchmarks and note some quality concerns with the NVFP4 quantization. **Sentiment:** 35% positive · 40% mixed · 25% skeptical **The case for** - Smaller, efficient models enable qualitatively different user experiences beyond just being cheaper versions of frontier models. - NVIDIA's hardware-selling incentive gives it a durable, long-term motivation to keep releasing open-weight models. - The model runs well on Apple Silicon via MLX, making it accessible on consumer hardware. - The fully open training pipeline, recipes, and partial training data make it uniquely reproducible among competitive models. - Speed gains from efficient models can unlock new categories of applications, not just cost savings. **The pushback** - NeMo Switchyard's README explicitly labels it 'experimental, not for production use,' contradicting the press release's deployment messaging. - Benchmark scores trail Qwen 3.6 35B-A3B by roughly 20% across most tasks despite the same 3B active parameters. - Prompt/KV caching is model-specific and breaks when switching between models, raising real questions about the router's practical cost savings. - The NVFP4 quantization causes a notable SWE-bench score drop compared to the BF16 version. - Competing models like Meta's 30B dense offering may outperform Nemotron Lightning on quality benchmarks, and Qwen updates are imminent. **By community** - hackernews (mixed): Engaged but divided — optimistic about the small/efficient model trend in principle, yet skeptical of Switchyard's production readiness, caching claims, and benchmark honesty. **Hottest debate:** NVIDIA Nemotron 3.5 Lightning and NeMo Switchyard Deliver Faster, Smarter, More Efficient Agentic AI: Whether small, efficient models represent a genuine architectural leap or are merely a temporary workaround until hardware and large-model scaling catch up. **Open questions** - How does NeMo Switchyard handle KV/prompt cache continuity when routing a session across different models? - Will a Qwen 3.8 35B-A3B MoE follow, and how much will it improve over 3.6? - What is the minimum VRAM required to run Nemotron 3.5 Lightning at Q4 quantization on consumer hardware? - Is the OpenRouter free endpoint serving the BF16 or NVFP4 variant? - Why does the README say 'not for production use' while the press release encourages deployment? **Highlights** > I think finding significant efficiency gains with LLMs and the like may lead to qualitatively better products. Looking at people's experiences to DSV4F makes me believe that even more than before too. I don't think people are realizing that speed can allow for categorically different user experiences that are more than just "worse than frontier capabilities but faster". > — [cootsnuck on hackernews](https://news.ycombinator.com/item?id=49264987) > Not by the looks of it, but it got me thinking, currently in the middle of Level1Techs coverage on the model and switchyard and he mentions "how easy it is to customize it". Fully admitting that I haven't yet read the docs, my issue with that is "we can train LORAs for 35B as well, why use this (according to benchmarks) worse model for customization instead of a slightly bigger better one?" Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp). Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs). > — [sleepyeldrazi on hackernews](https://news.ycombinator.com/item?id=49265159) > Runs fast on my Mac - ~100 tokens/second - but it's a bit of an over-thinker. It sketched out four different SVGs in the reasoning trace before returning a (bad) pelican riding a bicycle: https://tools.simonwillison.net/markdown-svg-renderer#url=ht... I used LM Studio and this model file: https://lmstudio.ai/models/nvidia/nemotron-3.5-lightning I got a pretty solid (and reasonably fast) result for this simply coding agent activity (with simonw/llm-coding-agent): llm code -m lmstudio/nvidia/nemotron-3.5-lightning \ 'how does auth work?' https://gist.github.com/simonw/a8741f79280cedc86bcb8d90edfb4... - took just under a minute. > — [simonw on hackernews · 2 comments](https://news.ycombinator.com/item?id=49261268) > That SWEBench drop going from bf16 to fp4 is massive > — [piyh on hackernews](https://news.ycombinator.com/item?id=49260107) > The Nemotron 3 family uses Mamba-2 because that is what we had validated when we started building the family. Changing the SSM backbone is a major revision, not a dot release. At least for us. > — [bcatanzaro on hackernews](https://news.ycombinator.com/item?id=49262066) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49263340) · 124 points · 63 comments - [hackernews](https://news.ycombinator.com/item?id=49257947) · 101 points · 29 comments ## Similar posts on daily.dev - [Nvidia launches a smaller, faster Nemotron model and a router to put it to work](https://daily.dev/posts/nvidia-launches-a-smaller-faster-nemotron-model-and-a-router-to-put-it-to-work-adgtw6klw) · The New Stack · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#nvidia](https://daily.dev/tags/nvidia), [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/nvidia-releases-nemotron-3-5-lightning-and-nemo-switchyard-for-ai-agent-workloads-8kkbicyjr) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"NVIDIA releases Nemotron 3.5 Lightning and NeMo Switchyard for AI agent workloads","url":"https://daily.dev/posts/nvidia-releases-nemotron-3-5-lightning-and-nemo-switchyard-for-ai-agent-workloads-8kkbicyjr","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/nvidia-releases-nemotron-3-5-lightning-and-nemo-switchyard-for-ai-agent-workloads-8kkbicyjr"},"datePublished":"2026-08-11T13:08:18.157Z","dateModified":"2026-08-17T18:20:22.897Z","description":"NVIDIA has released two tools targeting AI agent workloads: Nemotron 3.5 Lightning, a 30B mixture-of-experts model with only 3B active parameters at inference,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5675ffc5845b99cb93ebdba29bb48d15?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5675ffc5845b99cb93ebdba29bb48d15?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/nvidia-releases-nemotron-3-5-lightning-and-nemo-switchyard-for-ai-agent-workloads-8kkbicyjr","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,ai-agents,nvidia,ai-inference","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"NVIDIA releases Nemotron 3.5 Lightning and NeMo Switchyard for AI agent workloads"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/nvidia-releases-nemotron-3-5-lightning-and-nemo-switchyard-for-ai-agent-workloads-8kkbicyjr#faq","mainEntity":[{"@type":"Question","name":"How much faster is Nemotron 3.5 Lightning compared to Qwen3 35B on agent benchmarks?","acceptedAnswer":{"@type":"Answer","text":"Nemotron 3.5 Lightning completes 10,000 tasks 30% faster than Qwen3 35B at comparable accuracy, hitting 86% on PinchBench. NVIDIA also claims up to 4x faster token generation versus similar open models. The model achieves this using speculative decoding with multi-token prediction, NVFP4 quantization, and two draft models (DSpark and DFlash), while keeping only 3B of its 30B parameters active at inference. Teams benchmarking open models for agent workloads track comparisons like this on daily.dev."}},{"@type":"Question","name":"How much can NeMo Switchyard reduce AI agent inference costs in practice?","acceptedAnswer":{"@type":"Answer","text":"NeMo Switchyard's escalation router cut costs by 74% in LangChain testing with only a ~6-point accuracy drop, achieved by routing just 7% of calls to the frontier model. In Cognition's Devin Desktop deployment, staged routing reached 50.6% accuracy at $3.11 mean cost — within 2.8 percentage points of frontier accuracy at roughly 28% lower cost. NVIDIA also reports benchmark completion costs around one-third of using Opus 4.8 alone. Developers optimizing agent pipeline spend find the latest cost-vs-accuracy tradeoffs on daily.dev."}},{"@type":"Question","name":"What hardware can run Nemotron 3.5 Lightning locally?","acceptedAnswer":{"@type":"Answer","text":"Nemotron 3.5 Lightning runs on consumer hardware including the GeForce RTX 5090, DGX Spark, Jetson, and OEM GB10 systems, and scales to data centers and cloud. Deployment is supported through vLLM, Ollama, llama.cpp, LM Studio, and Unsloth, with NVFP4 and GGUF formats available. Weights, training data, and recipes are fully open under the OpenMDW-1.1 license. Developers self-hosting open models for agent workloads follow hardware and deployment news on daily.dev."}}]} ```