OpenAI audited SWE-Bench Pro, a widely used coding benchmark, and found that approximately 30% of its 731 public tasks are broken. The audit used a pipeline combining Codex-based investigator agents and reviews by five experienced software engineers per flagged task. Four main failure categories were identified: overly strict tests that enforce specific implementation details, underspecified prompts that omit requirements, low-coverage tests that allow incomplete fixes to pass, and misleading prompts that contradict test expectations. Human reviewers were more conservative than agents, finding more broken tasks and more overlapping issues per task. OpenAI retracts its earlier recommendation to adopt SWE-Bench Pro and calls on the evaluation community to build new benchmarks designed specifically for testing model capabilities rather than repurposing open-source pull request history.
