GitHub - jundot/omlx: LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

oMLX is an open-source LLM inference server built specifically for Apple Silicon Macs, featuring continuous batching and a tiered KV cache system that spans hot RAM and cold SSD storage. It supports text LLMs, vision-language models, OCR, embeddings, and rerankers, all managed through a native macOS menu bar app or web admin dashboard. Key features include multi-model serving with LRU eviction and model pinning, OpenAI/Anthropic API compatibility, MCP tool integration, Claude Code optimization, and one-click integrations with tools like OpenCode and Codex. It can be installed via a .dmg, Homebrew, or from source, and requires macOS 15+ with Apple Silicon (M1–M4).