How I built a sub-500ms latency voice agent from scratch
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A developer built a custom voice agent orchestration layer from scratch in roughly one day, achieving ~400ms end-to-end latency — 2× faster than Vapi's equivalent setup. The post covers the full architecture: using Silero VAD for turn detection, Deepgram Flux for streaming transcription and turn-taking, Groq's llama-3.3-70b for ultra-low TTFT (~80ms), and ElevenLabs for TTS with pre-warmed WebSocket connections. Key lessons include: pipelining STT→LLM→TTS as a streaming chain rather than sequential steps, handling barge-ins by cancelling all in-flight requests simultaneously, and treating geographic co-location of services as a first-class design parameter. Deploying to Railway EU and using regional endpoints for Twilio, Deepgram, and ElevenLabs cut latency from 1.7s to ~790ms, and switching to Groq pushed it down to ~400ms. Source code is available on GitHub.
