Building AI agents for 127 million customers
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Nubank's engineering team shares five hard-won lessons from building production AI agents for 127 million customers. Key insights include: adopting an evals-first approach using TNPS and LLM-as-a-judge instead of traditional NPS; defining agents through the ReAct paradigm with distinct prompt, tool, and data layers; using automated prompt optimization (DSPy, Japa) instead of hand-written prompts; avoiding fine-tuning until frontier model capabilities are exhausted; and moving deterministic business logic into composite tools rather than relying on LLM reasoning. The post also covers automated simulation and red-teaming for pre-production validation.
