AI red teaming has evolved from a niche ML security practice into a broad discipline covering cybersecurity, safety, misinformation, and autonomous agent risks. Unlike traditional penetration testing, AI systems are probabilistic — the same attack may succeed only occasionally — requiring repeated testing under varying conditions. Threat models now include not just nation-state actors but ordinary users who discover jailbreaks through curiosity. Teams must test the entire AI stack (APIs, databases, workflows), not just the model itself. Agentic AI raises the stakes further: agents that execute real-world actions create operational risks beyond mere bad outputs. Microsoft, Anthropic, OpenAI, Google, and Nvidia all maintain dedicated AI red teams, and the field is converging with traditional cybersecurity as agentic systems combine both risk categories.
