Voice & Real-Time Agents
Real-time voice agents = STT (speech-to-text) → LLM → TTS (text-to-speech) stitched into a low-latency loop (or, increasingly, a single speech-to-speech model). The "platform" layer (Vapi/Retell/Bland) handles telephony, turn-taking, and orchestration; the "infra" layer (LiveKit/Pipecat) gives you the transport/pipeline to build your own; and the "model" layer (ElevenLabs/Deepgram/Cartesia + realtime APIs) supplies the ears and voice. All 🔄. Pricing here is per-minute and shifts often.
Platforms (build a phone/voice agent fast)
| Platform | Best at | Watch out | Price |
|---|---|---|---|
| Vapi | Fast time-to-production with pluggable STT/LLM/TTS, function calling, and telephony; big ecosystem | Per-minute cost stacks (platform + each model + telephony), so model your unit economics | Usage-based, ~$0.05+/min plus provider costs 🔄 |
| Retell AI | Natural turn-taking and call reliability for support/sales | None | Usage-based per minute 🔄 |
| Bland AI | High-volume outbound/inbound calling with an all-in-one stack | More closed than Vapi or Retell | Usage-based 🔄 |
Also: Synthflow, Vocode (open source).
Infrastructure (build your own pipeline)
| Tool | Best at | Key fact |
|---|---|---|
| LiveKit | Scalable real-time audio/video transport and building custom voice agents (self-host or cloud); the transport backbone many voice apps and some labs' voice features run on | Open-source WebRTC platform + an Agents framework; LiveKit Cloud is usage-priced |
| Pipecat | Composing the STT→LLM→TTS pipeline with full control and vendor-swappable components | Open-source Python framework (BSD), originated at Daily |
| Ultravox | Lower-latency, more natural voice by feeding audio straight into an LLM-derived model | Open-weight speech-to-speech that skips separate STT; open weights + hosted API, per-minute 🔄 |
Voices & ears (TTS / STT / realtime models)
| Tool | Best at | Watch out | Price |
|---|---|---|---|
| ElevenLabs | Natural, expressive TTS, voice cloning, dubbing, and a full Agents platform; broad language coverage | Premium pricing; voice-cloning consent/ethics matter | Free tier + paid from ~$5/mo (character-metered) 🔄 |
| Deepgram | Fast, accurate, cheap streaming STT (and now TTS) at scale (Nova-series) | None | Usage-based per minute/hour; free credits to start 🔄 |
| Cartesia | The fastest realtime TTS (Sonic, state-space models) with tiny time-to-first-audio | None | Usage-based, free tier 🔄 |
Realtime APIs: OpenAI Realtime API (speech-to-speech via GPT-4o-realtime/successors), Google Gemini Live API, AssemblyAI (STT with speech understanding).
Latency budget rule: conversational agents feel natural under ~800ms round-trip. Speech-to-speech models (Ultravox, OpenAI Realtime) cut the STT/TTS hops. Classic pipelines win on component choice and cost control.