Pendo's Chief AI Officer describes how they built Novus, an AI product agent that monitors user behavior via session replays and behavioral analytics, diagnoses usability issues, and generates code fixes. LangSmith was used as the core observability platform throughout development and production, enabling the team to trace every agent decision, debug unexpected tool calls, and catch 60% of AI problems before customers reported them. A key insight discovered through traces was that the agent was using either analytics data or code context but rarely both — a prompt tuning fix that became central to Novus's value proposition. The system achieves a 90%+ success rate on PM-reviewed evaluations and was shipped to production in days.
