JFrog has open-sourced agent-belt, a CLI-based eval harness for AI coding agents. It tests agents like Claude Code, Cursor, Copilot, Codex, and others by running their actual CLIs as subprocesses against real workspaces with MCP servers wired up. Scenarios are JSON-defined multi-turn test cases with deterministic rule checks (tool sequences, file modifications, git diffs) and LLM-based scoring using a separate judge model. Key features include k-trial variance reduction, scenario families for input robustness, multi-judge consensus voting, Docker sandboxing, CI integration with exit codes, and four export formats. The tool addresses the gap where existing eval frameworks test models or wrapped functions rather than the actual agent CLI a user installs and runs.
