A deep dive into the design patterns behind cybersecurity evaluation benchmarks for AI agents. The post identifies four core primitives shared across benchmarks: a sandboxed target (Docker containers), inputs that control difficulty (zero-day vs. one-day scenarios), tools (bash, debuggers, websearch), and a grader for deterministic outcome checking. Seven benchmarks are analyzed in detail — Cybench (CTF tasks), CVE-Bench (real-world CVEs), CyberGym (PoC generation for C/C++ memory bugs), ExploitGym (turning PoCs into full exploits), ExploitBench (V8 engine exploitation ladder), MHBench (multi-host red-team operations), and SCONE-Bench (smart contract exploitation). Key findings include: top models achieve 10–22% success rates on most tasks, system framework design matters more than model choice (Incalmo vs. ExpertPromptShell), security defenses sharply reduce exploit success, and partial credit via subtasks provides more granular capability measurement.
