Stripe built a benchmark to evaluate whether AI agents can autonomously complete real-world Stripe API integrations end to end. The benchmark includes 11 diverse environments spanning backend-only tasks, full-stack tasks, and gym problem sets, using a goose-based agent harness with MCP tools for terminal, browser, and Stripe documentation access. Results showed Claude Opus 4.5 achieved 92% on full-stack tasks and GPT-5.2 scored 73% on gym sets. Agents surprised researchers by navigating UIs, debugging live issues, and handling underdocumented behavior. Key failure modes included mishandling ambiguous situations (e.g., treating 400 errors as success) and getting stuck during browser interactions. The benchmark is open-sourced to help the community improve agentic tooling for API integrations.
