Arm SME2 and Google AI Edge together enable high-performance on-device generative AI inference on CPUs. The post walks through converting Stability AI's stable-audio-open-small PyTorch model into a mixed-precision FP16/INT8 format using LiteRT-Torch and AI Edge Quantizer, then deploying it via LiteRT with automatic KleidiAI acceleration through XNNPACK. Results show 3x performance improvement and 4x memory reduction in the DiT submodule, with SME2 delivering over 2x speedup versus NEON on Android. The pipeline covers model conversion, quantization error visualization with Model Explorer, and C++ runtime inference using the CompiledModel API.
Table of contents
Google AI Edge: A seamless path from PyTorch to siliconConvert : Convert from PyTorch to .tflite with LiteRT TorchOptimize: Optimize with Model Explorer and AI Edge QuantizerDeploy: High-performance inference with LiteRT via XNNPack & KleidiAIResults : Faster, smaller, and high-quality audio generation with a lower footprintReady to learn more?Developer resources79 Impressions