--- title: Building a 3D AI assistant using React Native and... description: A deep technical walkthrough of building AvatarAssist, a Talking Tom-inspired 3D AI assistant in React Native. The squirrel character is rendered with... canonical: https://daily.dev/posts/building-a-3d-ai-assistant-using-react-native-and-gpt-realtime-zomhbknb4 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Building a 3D AI assistant using React Native and GPT-Realtime | daily.dev og:description: A deep technical walkthrough of building AvatarAssist, a Talking Tom-inspired 3D AI assistant in React Native. The squirrel character is rendered with... og:url: https://daily.dev/posts/building-a-3d-ai-assistant-using-react-native-and-gpt-realtime-zomhbknb4 og:image: https://api.daily.dev/og/posts/ZomHBKNb4.png og:image:alt: Building a 3D AI assistant using React Native and GPT-Realtime og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Building a 3D AI assistant using React Native and GPT-Realtime **[Nitro Modules](https://daily.dev/sources/margelo)** · 29 min read · 2 upvotes · 2 comments ## Summary A deep technical walkthrough of building AvatarAssist, a Talking Tom-inspired 3D AI assistant in React Native. The squirrel character is rendered with react-native-filament (a port of Google's Filament engine), animated via shared values and render callbacks running on the UI thread, and voiced by OpenAI's gpt-realtime over a native WebSocket for low-latency streamed audio instead of a traditional LLM-to-TTS pipeline. Lip sync is approximated by measuring RMS loudness of audio samples rather than true phoneme mapping. Tool calling lets the assistant write real events to the device calendar, with careful handling of local time and ISO 8601 formatting to avoid UTC mistakes. A benchmark comparing on-device LLMs against the cloud-based gpt-realtime pipeline shows the cloud approach is roughly 10x faster to first speech (574ms vs 6,130ms), uses far less CPU, and is more accurate at tool calls and calendar times, explaining why the team chose cloud inference over on-device models. The piece also covers startup performance tricks like native splash screen dismissal and trimming unused animation clips, plus native UI touches like liquid glass buttons and keyboard-aware layout. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much faster is OpenAI's gpt-realtime compared to running a local on-device LLM plus TTS pipeline on a phone? Cloud-based gpt-realtime delivered time to first speech of 574ms versus 6,130ms for the best on-device model tested, roughly ten times faster. On-device also used 238% CPU during a turn versus 37% for cloud, got calendar times correct in 0 of 5 tests versus 4 of 5, and added 1.0GB to app size versus none, though it worked offline. _Weigh cloud versus on-device AI tradeoffs like these before picking an architecture, tracked on daily.dev._ ### How can I get audio playback speed correct when using PCM16 audio from an OpenAI realtime API in a React Native app? The AudioContext sample rate must exactly match the rate requested in the session config, for example 24000 for gpt-realtime's audio.output.format. If they mismatch, playback still works but plays at the wrong speed, making the voice sound too high-pitched, since nothing throws an error to flag the mismatch. _Debugging subtle audio glitches like this is easier when developers share fixes on daily.dev._ ### Why does an LLM tool call sometimes write the wrong time to a calendar event even when scheduling logic looks correct? Language models have no built-in sense of current time and will guess a date based on training data, and they also default to appending a trailing 'Z' UTC marker even when local time was intended, silently shifting event times by hours. The fix is passing the current local time and UTC offset in the system instructions and giving an explicit example of the desired ISO 8601 local-time format. _Catch timezone gotchas like this before they ship by following AI tool-calling patterns on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@rizzdev** · 1 upvotes > The 3D squirrel casts a PNG because the forest is a looping video **@agustinbarrientos** · 1 upvotes > That live-amplitude mouth animation is a clever touch, nice build! ## Similar posts on daily.dev - [How I Made My Own Offline Voice Assistant](https://daily.dev/posts/how-i-made-my-own-offline-voice-assistant-lvephkfkw) · Medium · 0 upvotes · 0 comments - [Build a real-time hotel booking voice agent](https://daily.dev/posts/build-a-real-time-hotel-booking-voice-agent-ntorwqf8n) · Daily Dose of Data Science \| Avi Chawla \| Substack · 1 upvotes · 0 comments --- Tags: [#webdev](https://daily.dev/tags/webdev), [#3d](https://daily.dev/tags/3d), [#react-native](https://daily.dev/tags/react-native) [View this post on daily.dev](https://daily.dev/posts/building-a-3d-ai-assistant-using-react-native-and-gpt-realtime-zomhbknb4) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Building a 3D AI assistant using React Native and GPT-Realtime","url":"https://daily.dev/posts/building-a-3d-ai-assistant-using-react-native-and-gpt-realtime-zomhbknb4","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/building-a-3d-ai-assistant-using-react-native-and-gpt-realtime-zomhbknb4"},"datePublished":"2026-08-23T12:22:09.367Z","dateModified":"2026-09-14T06:42:57.032Z","description":"A deep technical walkthrough of building AvatarAssist, a Talking Tom-inspired 3D AI assistant in React Native. The squirrel character is rendered with...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fd5536608ad0feb7286f2ea162fd36ad?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/fd5536608ad0feb7286f2ea162fd36ad?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Nitro Modules","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Nitro Modules","logo":"https://media.daily.dev/image/upload/s--5qrQGNNm--/f_auto,q_auto/v1787487502/logos/margelo?_a=BAMAMicg0","url":"https://daily.dev/sources/margelo"},"commentCount":2,"discussionUrl":"https://daily.dev/posts/building-a-3d-ai-assistant-using-react-native-and-gpt-realtime-zomhbknb4","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":2}],"keywords":"webdev,3d,react-native","timeRequired":"PT29M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Nitro Modules","item":"https://daily.dev/sources/margelo"},{"@type":"ListItem","position":3,"name":"Building a 3D AI assistant using React Native and GPT-Realtime"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/building-a-3d-ai-assistant-using-react-native-and-gpt-realtime-zomhbknb4","comment":[{"@type":"Comment","text":"The 3D squirrel casts a PNG because the forest is a looping video","datePublished":"2026-08-23T14:32:03.237Z","url":"https://daily.dev/posts/ZomHBKNb4#c-uBaivk9n6","author":{"@type":"Person","name":"Andrew","url":"https://daily.dev/rizzdev","image":"https://media.daily.dev/image/upload/s--35vRfXAA--/f_auto/v1785413727/avatars/avatar_fQ7ttCKtPuu6tDvcgQXv5?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"That live-amplitude mouth animation is a clever touch, nice build!","datePublished":"2026-08-27T15:43:10.080Z","url":"https://daily.dev/posts/ZomHBKNb4#c-6KVrlwO91","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/building-a-3d-ai-assistant-using-react-native-and-gpt-realtime-zomhbknb4#faq","mainEntity":[{"@type":"Question","name":"How much faster is OpenAI's gpt-realtime compared to running a local on-device LLM plus TTS pipeline on a phone?","acceptedAnswer":{"@type":"Answer","text":"Cloud-based gpt-realtime delivered time to first speech of 574ms versus 6,130ms for the best on-device model tested, roughly ten times faster. On-device also used 238% CPU during a turn versus 37% for cloud, got calendar times correct in 0 of 5 tests versus 4 of 5, and added 1.0GB to app size versus none, though it worked offline. Weigh cloud versus on-device AI tradeoffs like these before picking an architecture, tracked on daily.dev."}},{"@type":"Question","name":"How can I get audio playback speed correct when using PCM16 audio from an OpenAI realtime API in a React Native app?","acceptedAnswer":{"@type":"Answer","text":"The AudioContext sample rate must exactly match the rate requested in the session config, for example 24000 for gpt-realtime's audio.output.format. If they mismatch, playback still works but plays at the wrong speed, making the voice sound too high-pitched, since nothing throws an error to flag the mismatch. Debugging subtle audio glitches like this is easier when developers share fixes on daily.dev."}},{"@type":"Question","name":"Why does an LLM tool call sometimes write the wrong time to a calendar event even when scheduling logic looks correct?","acceptedAnswer":{"@type":"Answer","text":"Language models have no built-in sense of current time and will guess a date based on training data, and they also default to appending a trailing 'Z' UTC marker even when local time was intended, silently shifting event times by hours. The fix is passing the current local time and UTC offset in the system instructions and giving an explicit example of the desired ISO 8601 local-time format. Catch timezone gotchas like this before they ship by following AI tool-calling patterns on daily.dev."}}]} ```