--- title: "How I built a sub-500ms latency voice agent from scratch" url: https://daily.dev/posts/how-i-built-a-sub-500ms-latency-voice-agent-from-scratch-neao8lbjr source_url: https://www.ntik.me/posts/voice-agent type: article source: "Hacker News" published: 2026-03-03T00:03:30.776Z updated: 2026-03-03T00:03:57.965Z tags: ["speech-recognition", "text-to-speech"] reading_time: 14 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How I built a sub-500ms latency voice agent from scratch **[Hacker News](https://daily.dev/sources/hn)** · 14 min read · 0 upvotes · 0 comments ## Summary A developer built a custom voice agent orchestration layer from scratch in roughly one day, achieving ~400ms end-to-end latency — 2× faster than Vapi's equivalent setup. The post covers the full architecture: using Silero VAD for turn detection, Deepgram Flux for streaming transcription and turn-taking, Groq's llama-3.3-70b for ultra-low TTFT (~80ms), and ElevenLabs for TTS with pre-warmed WebSocket connections. Key lessons include: pipelining STT→LLM→TTS as a streaming chain rather than sequential steps, handling barge-ins by cancelling all in-flight requests simultaneously, and treating geographic co-location of services as a first-class design parameter. Deploying to Railway EU and using regional endpoints for Twilio, Deepgram, and ElevenLabs cut latency from 1.7s to ~790ms, and switching to Groq pushed it down to ~400ms. Source code is available on GitHub. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [How I Made My Own Offline Voice Assistant](https://daily.dev/posts/how-i-made-my-own-offline-voice-assistant-lvephkfkw) · Medium · 0 upvotes · 0 comments - [One-Second Voice-to-Voice Latency with Modal, Pipecat, and Open Models](https://daily.dev/posts/one-second-voice-to-voice-latency-with-modal-pipecat-and-open-models-aejtkcqop) · Modal · 0 upvotes · 0 comments --- Tags: [#speech-recognition](https://daily.dev/tags/speech-recognition), [#text-to-speech](https://daily.dev/tags/text-to-speech) [View this post on daily.dev](https://daily.dev/posts/how-i-built-a-sub-500ms-latency-voice-agent-from-scratch-neao8lbjr)