A practical breakdown of automated red teaming costs for LLM-based AI agents using Promptfoo, based on internal R&D testing of a production AI sales agent called Omega. The post covers execution costs across major security frameworks (NIST, OWASP LLM Top 10, MITRE ATLAS, EU AI Act, GDPR, ISO 42001), showing full configurations ranging from ~$273 to ~$1,239 per run. It also presents a simplified configuration strategy that reduces costs by 80%+ (e.g., NIST from ~$400 to ~$80) with trade-offs in coverage confidence. Key limitations discussed include LLM-evaluating-LLM variability, tool immaturity, and the need for human expertise to interpret results. Step-by-step Promptfoo setup commands are included.
Table of contents
Automated AI security testing with Promptfoo: What we testedSecurity frameworks and real execution cost in AI security testingAI security tools: Full framework configurationHow to reduce automated AI security testing costsRunning Promptfoo redteam tests against an AI agent (Omega example)Limitations of AI security toolsWhat automated AI security testing is actually good forBonus: Live AI agent hacking demo and LLM security checklistSummary: QA reviewer observations from AI security testing178 Impressions