NVIDIA Nemotron 3.5 Lightning is a hybrid MoE model with 30B total parameters and only 3B active per token, designed for always-on agentic workloads. vLLM now supports it on day-0 with an OpenAI-compatible API, continuous batching, prefix caching, and three speculative decoding strategies: Multi-Token Prediction (MTP), DFlash (diffusion-based drafter), and DSpark (hybrid autoregressive + diffusion). Key upstream contributions to vLLM include DSpark integration, a quantized W4A16 draft head, removal of host-device syncs, a Hopper-optimized Humming backend replacing Marlin (yielding ~20% throughput gain), and ReplaySSM for Mamba2 layers. The model supports up to 1M token context, BF16 and NVFP4 checkpoints, controllable reasoning per request, and targets hardware from NVIDIA Jetson and DGX Spark to H100/H200/B200. Benchmarks show it completes agentic tasks up to 30% faster than Qwen3.6 35B at comparable accuracy.
Table of contents
TL;DR: About Nemotron 3.5 LightningRun High-Throughput Inference with vLLMAccelerate Long-Running Agentic Workflows with Speculative DecodingLocal Deployment on NVIDIA DGX SparkLocal Deployment on NVIDIA JetsonLeading Accuracy and Efficiency for Specialized Agent TasksSummaryAcknowledgementFAQsQuestions this post answers
What speculative decoding methods does NVIDIA Nemotron 3.5 Lightning support in vLLM?
Nemotron 3.5 Lightning supports three speculative decoding techniques in vLLM: Multi-Token Prediction (MTP), which uses built-in prediction heads to propose future tokens; DFlash, which uses a diffusion-based draft model to propose a full token block in parallel; and DSpark, a hybrid that blends autoregressive and diffusion-style drafting. DSpark delivers the best performance on DGX Spark, while running without speculative decoding maximizes throughput. Teams tuning agentic inference pipelines track speculative decoding trade-offs like these on daily.dev.
How much throughput improvement does Nemotron 3.5 Lightning offer over similarly sized open models?
Nemotron 3.5 Lightning delivers up to 4x higher throughput than similarly sized open models. This comes from its hybrid MoE architecture activating only 3B of 30B parameters per token combined with multi-token prediction. On agentic task benchmarks (PinchBench), it completes 10,000 tasks up to 30% faster than Qwen3.6 35B at comparable accuracy levels. Developers choosing between efficient agentic models for high-volume workloads find the latest comparisons on daily.dev.
What upstream changes did the Nemotron team contribute to vLLM for this release?
Five contributions landed upstream in vLLM: DSpark speculator integration, a W4A16-quantized DSpark draft head that reduces memory and per-step latency, removal of host-device syncs and async scheduling in the draft-and-verify loop, replacement of the Marlin backend with a Hopper-optimized Humming backend using W4A16 GEMM kernels (worth roughly 20% throughput), and ReplaySSM integration for Mamba2 state-space layers to reduce per-step overhead. Engineers contributing to or building on vLLM follow upstream changes like these on daily.dev.