A detailed technical breakdown of optimizations made to vLLM for Arm Neoverse-based CPUs, covering the full inference stack. Key improvements include switching to mimalloc as the default allocator (2.3× throughput gain), leveraging Arm LSE atomics in OpenMP to reduce synchronization overhead (9% throughput, 15% TPOT improvement), enabling oneDNN weight prepacking for dense layers (16% throughput, 60% TPOT improvement), and rewriting paged attention with BFMMLA SIMD instructions (4× faster attention, 12% throughput gain). INT8 W8A8 quantization delivers up to 88% higher throughput and 45% lower TPOT versus the optimized BF16 baseline, while INT8 W4A8 pushes further with up to 29% additional throughput gains. Combined, these changes yield up to 6.2× the October 2025 baseline throughput and a 7.8× TPOT speedup. Usability improvements include pre-built wheels and Docker images, chunked prefill, prefix caching, and expanded model support.
