Agent drift is real and your unit tests won’t catch it
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Agent drift describes the gap between how an AI agent behaves in testing versus production, where real users apply pressure that exposes scope creep, hallucination, and policy violations. ArkSim is an open source Python tool that addresses this by running profile-driven synthetic users against your agent in multi-turn conversations, then scoring each turn across helpfulness, coherence, relevance, faithfulness, and goal completion. The tool supports Python and HTTP connectors, generates HTML reports with named failure types, and integrates into CI/CD via threshold gates that exit with code 1 on failure. Three bundled examples cover scope enforcement (insurance), faithfulness under changing data (e-commerce), and goal completion (open-ended assistants).
