Google AI Edge's LiteRT runtime enables deploying Gemma language models directly on Raspberry Pi 5 for fully offline, real-time edge AI applications. On a Raspberry Pi 5, LiteRT-LM achieves 99 tokens/sec prefill and 9 tokens/sec decode for Gemma 4 E2B with a 1432 MB peak memory footprint, translating to ~27.3 characters/sec (~300 wpm) end-to-end generation. The post demonstrates a dual CPU/GPU architecture where LLM inference runs on the ARM Cortex-A76 CPU while vision and audio models offload to the VideoCore VII GPU via LiteRT's WebGPU (Vulkan) backend. A Reachy Mini robot demo illustrates real-time perception and response entirely offline. Setup is straightforward via `pip install litert-cli`, with models pulled from the LiteRT Hugging Face Community. Hailo AI accelerator support is also announced as coming soon.
Table of contents
Discover the Agentic Capability of GemmaGemma Performance on Raspberry Pi CPUExecute on Raspberry Pi GPU with LiteRTDeep Dive: Reachy Mini Pipeline Powered by LiteRTAgentic Coding with LiteRT on Raspberry PiAn Ultra-Lean Binary Footprint for IoT DevicesRunning Your First ModelWhat’s NextAcknowledgementsQuestions this post answers
What inference performance does Gemma 4 E2B achieve on a Raspberry Pi 5 with LiteRT-LM?
On a Raspberry Pi 5, LiteRT-LM running Gemma 4 E2B achieves 99 tokens/sec for prefill and 9 tokens/sec for decode, with a peak memory footprint of 1432 MB. Thanks to the model's efficient tokenizer averaging ~4.2 characters per token, end-to-end generation reaches ~27.3 characters/sec, roughly 300 words per minute — twice the speed of normal human speech. Developers targeting Raspberry Pi robotics or voice applications track LiteRT and Gemma performance updates on daily.dev.
How do I install and run a Gemma model on Raspberry Pi 5 using LiteRT CLI?
Install LiteRT CLI with `pip install litert-cli`, then set your Hugging Face token via `export HUGGING_FACE_HUB_TOKEN=<token>` and run `litert lm run --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm gemma-4-E2B-it.litertlm`. Models are pulled directly from the LiteRT Hugging Face Community, and the CLI handles conversion, quantization, and inference in a single unified command set. Engineers shipping edge AI prototypes find the latest LiteRT tooling news on daily.dev before it hits the docs.
How does LiteRT split workloads between CPU and GPU on Raspberry Pi 5 for real-time AI?
LiteRT uses a dual-processing architecture where LLM inference runs on the quad-core ARM Cortex-A76 CPU (~153.6 GFLOPS FP32, ~2.0 TOPS INT8), while continuous vision and audio models are offloaded to the Broadcom VideoCore VII GPU via LiteRT's WebGPU (Vulkan) backend through ML Drift. This preserves CPU cycles for high-priority tasks and improves thermal efficiency. Robotics developers balancing compute budgets on constrained hardware follow edge AI architecture discussions on daily.dev.