28.9M-parameter LLM runs locally on ESP32-S3 at 9 tokens/s

Developer Slava S. has managed to run a 28.9M-parameter LLM entirely on an ESP32-S3 microcontroller at ~9 tokens/second, without relying on any cloud backend. The key insight is using Google's Per-Layer Embeddings technique (from Gemma 3n/4): since most parameters reside in a large embedding table (~25M rows), they can be stored in slow 16MB flash memory rather than fast SRAM. Only the ~450 bytes needed per token are loaded into the 512KB SRAM at inference time. The model (14.9MB at 4-bit quantization) was trained on TinyStories and can only generate short stories — it cannot answer questions or follow instructions. The project runs on an ~$8 ESP32-S3 board with 8MB PSRAM and 16MB flash, connected to a small I2C display.