Slack's engineering team ran 200+ agentic E2E test executions using three approaches: Playwright MCP, Playwright CLI, and AI-generated Playwright tests, powered by Claude models. Key findings: Playwright MCP was the most reliable (0–12% failure rate) and cost-efficient (~40 turns per run), while CLI had higher failure rates (12–20%) due to execution layer issues. Generated Playwright tests were fastest (~3 min) but degraded significantly on complex flows (~48% failure rate). Agent runs cost $15–30 each due to context accumulation across turns, not model output. Only ~20% of agentic runs followed identical action sequences, demonstrating goal-oriented adaptability. The conclusion: agentic testing doesn't replace deterministic E2E tests but adds an exploratory layer at the top of the testing pyramid, best suited for debugging, reproducing production bugs, and validating complex UI behaviors rather than high-frequency CI regression.
