TanStack AI now supports realtime voice conversations with a provider-agnostic architecture covering OpenAI Realtime (WebRTC) and ElevenLabs (WebSocket). It uses a token/connection split so API keys stay server-side while audio streams directly from browser to provider. A React hook, useRealtimeChat, manages connection lifecycle, audio capture/playback, voice activity detection (server, semantic, manual modes), client-side tool execution via toolDefinition(), multimodal input (text and images), audio visualization data, streaming transcripts, and interruption handling. A framework-agnostic RealtimeClient class is also provided for non-React usage. Example use cases include voice-controlled dashboards, multimodal support, language tutoring, accessibility interfaces, and field data collection.
Table of contents
The Architecture: Tokens on the Server, Connections on the Client #Server Side: One Function, Any Provider #Client Side: useRealtimeChat #What mode Tells You #Voice Activity Detection: Three Modes #Tools: The Same Definitions, Running on the Client #Multimodal Input: Text and Images Alongside Voice #Audio Visualization #Live Transcripts #Interruptions #Session Configuration #Switching Providers at Runtime #Using the Client Directly (Without React) #Ideas: What You Could Build #Getting Started #Questions this post answers
How do I add realtime voice chat to a React app using TanStack AI?
Use the useRealtimeChat hook from @tanstack/ai-react, passing a getToken function and an adapter like openaiRealtime() or elevenlabsRealtime(). The hook manages connection lifecycle, audio capture/playback, voice activity detection, transcripts, and tool execution automatically. Support covers OpenAI Realtime via WebRTC and ElevenLabs via WebSocket, with a token generated server-side to keep API keys safe. Developers wiring up voice interfaces can track new library capabilities like this on daily.dev.
How do you keep your API key safe when building realtime voice chat that streams audio directly from the browser to a provider like OpenAI Realtime?
Use a token/connection split: the server generates a short-lived token via a token adapter (such as openaiRealtimeToken or elevenlabsRealtimeToken) and returns only that token to the client, while the actual audio stream connects directly from browser to provider without passing through your server. This avoids exposing the raw API key while still supporting WebRTC, which cannot be proxied through a backend. Teams evaluating voice AI architectures can compare token-based approaches like this via daily.dev.
What voice activity detection modes does TanStack AI support for realtime voice chat?
Three modes are supported: server VAD, where the provider detects speech server-side and is the default; semantic VAD, available only for OpenAI, which uses the model's understanding of conversation flow with a semanticEagerness setting of low, medium, or high; and manual mode, where the developer controls start and stop via startListening() and stopListening(), useful for push-to-talk interfaces. daily.dev helps developers comparing VAD strategies stay current on voice interface tooling.