Three Years of Building Agents in Production (Part 1)

A mabl engineer shares three years of lessons building AI agents for software testing in production. Starting with failed PaLM experiments in 2023, the team learned to work with LLM strengths rather than against them, pivoting to LLM-as-a-judge for GenAI Assertions, implementing RAG with Gemini 2 for context memory, and building cross-platform state management. Key insights include: early agents couldn't reliably navigate simple UIs, semantic grouping worked better than word similarity matching, static test datasets don't scale for probabilistic AI features, and rigid Chain-of-Thought prompts backfired when upgrading to Gemini 2.5 because the model reasoned about the CoT structure instead of the actual problem. The authoring agent was rewritten five times. The post ends with the team recognizing they needed a unified agent framework rather than isolated smart features.