A hands-on exploration of building a fully on-device voice-to-structured-action feature on Android, replacing a cloud-based Gemini/Firebase AI Logic approach. The implementation uses whisper.cpp (a C/C++ runtime for OpenAI's Whisper model) via JNI for local speech transcription, and a deterministic Kotlin rule-based parser instead of an LLM for extracting structured payment intents. Key topics covered include Gradle NDK/CMake setup, audio capture and PCM-to-float conversion, model download and SHA-256 verification, native memory lifecycle management, and the trade-offs between cloud LLM flexibility versus on-device privacy, offline support, and determinism.
Table of contents
Whisper as a black boxFeeding the model: audio captureModel managementDeterministic parser238 Impressions