Ello shares the architectural decisions behind building a real-time AI tutor for children ages 4-9, where sub-second response time is non-negotiable. Key innovations include: replacing the standard LLM tool loop with a custom streaming harness that parses and executes actions while the model is still generating; an asynchronous planner agent that reasons about pedagogy in the gaps while a converser handles real-time interaction; pre-generating responses to predicted child answers on branched trajectories; and running a safety classifier in parallel with an eager response model so safety checks never add latency. The post explains the tradeoffs of each approach, including cost, observability overhead, and occasional mispredictions.
187 Impressions