Traditional A/B testing playbooks are becoming obsolete in the AI era due to four major shifts: collapsing UI surfaces, accelerating product velocity, AI-driven personalization replacing manual tweaks, and rising per-usage costs from LLM tokens. The author argues teams should stop wasting engineering resources on minor UI optimizations, take bigger bets especially on monetization, and run experiments for 1-2 months rather than 2 weeks to capture long-term cohort effects. A contrarian point is also made: not everything needs to be tested — AI has already validated many best practices (like showing paid features to free users), so teams should adopt those as defaults and reserve experimentation for genuinely high-impact, system-level changes like freemium boundaries and credit systems.
