Separating signal from noise in coding evaluations
OpenAI
OpenAI identifies significant flaws in the SWE-Bench Pro coding benchmark, revealing that approximately 30% of its tasks are broken and unreliable.Summary
OpenAI conducted an audit of the SWE-Bench Pro coding benchmark to ensure model capabilities are measured accurately. Using an automated pipeline supplemented by human software engineers, they discovered that roughly 30% of the benchmark's tasks are flawed due to overly strict tests, underspecified prompts, low-coverage tests, or misleading requirements. As a result, OpenAI has retracted its recommendation for using SWE-Bench Pro, emphasizing the need for more rigorous, developer-curated benchmarks that better reflect actual model performance.
(Source：OpenAI)