Meta has introduced Muse Glimmer, an open-weight 30B-parameter model distilled from Muse Spark, designed for on-device agentic workflows. ExecuTorch now provides end-to-end support for running it on NVIDIA GPUs and Apple Silicon Macs. Key features include DFlash parallel diffusion-based speculative decoding (achieving 52.8% throughput improvement on M5 Pro), native GGUF/K-quant execution, 128K+ token context via a hybrid global/sliding-window KV-cache (only 13 of 52 layers are global), and multimodal text+image inputs. Prebuilt PTE artifacts are available on Hugging Face. The post details backend-specific optimizations for CUDA (CUDA graphs, packed K-quant kernels, FlashDecoding++) and MLX/Metal (RMSNorm, RoPE, SDPA lowered to native ops), plus an agentic harness supporting multi-session isolation, XML tool-call parsing, and Harmony chat templating. Video input and cross-session prefix sharing are planned for future releases.
Table of contents
Why ExecuTorch?QuickstartExecuting the PTEsUse cases enabledPerformanceUnder the HoodWhat’s nextReferencesQuestions this post answers
What performance improvement does DFlash speculative decoding give for Muse Glimmer on Apple Silicon?
DFlash speculative decoding achieves a 52.8% throughput improvement over solo (greedy) decoding for Muse Glimmer on an M5 Pro with 64 GiB RAM. Solo decoding reaches 21.6 tokens/second for text-image input, while DFlash reaches 33.0 tokens/second, with no quality regression reported. Engineers optimizing on-device LLM inference track speculative decoding benchmarks like these on daily.dev.
How does Muse Glimmer handle 128K token context efficiently on edge devices?
Muse Glimmer uses a hybrid KV-cache architecture: only 13 of its 52 layers are global-attention layers, while the remaining 39 use sliding-window attention. This limits KV-cache growth to the global layers, making 128K+ token contexts practical on edge hardware. ExecuTorch supports this architecture natively for both CUDA and MLX backends. Developers shipping long-context models to edge hardware find architecture trade-offs like this covered on daily.dev.
How does ExecuTorch export Muse Glimmer GGUF checkpoints for CUDA and Apple Silicon?
ExecuTorch exports directly from released GGUF checkpoints via its torch.export-based ahead-of-time stack. For CUDA, it compiles and autotunes Triton kernels for the detected GPU architecture, mapping Q4_K/Q5_K/Q6_K to packed INT4/5/6 with dp4a GEMV kernels. For MLX, it uses repacked or fused Metal kernels, and merges adjacent sub-blocks with identical scale/min into larger group sizes up to 128 when lossless. Teams deploying quantized models to multiple backends follow ExecuTorch developments on daily.dev.