Turns out a trusted coding benchmark, SWE-Bench Pro, was about 30% broken!

Hi, it's Shiichan! Today I found a story that might make you raise an eyebrow. A coding benchmark that a lot of people trusted turns out to have had way more problems than expected.

OpenAI News openai.com

What was announced?

According to OpenAI News, OpenAI ran its own audit of SWE-Bench Pro, a widely used coding benchmark, and published the results. OpenAI had previously pointed out design flaws and contamination issues in SWE-bench Verified and recommended the community move to SWE-Bench Pro instead. This time, they turned that same scrutiny on SWE-Bench Pro itself.

The results were striking: OpenAI's datapoint analysis pipeline flagged 200 tasks (27.4%) as likely broken, and a human annotation campaign identified 249 tasks (34.1%) as problematic. Based on these findings, OpenAI estimates that about 30% of SWE-bench Pro tasks are broken.

Why it matters

OpenAI reports results on a range of external and internal benchmarks with every model release, and these numbers feed into safety decisions made under OpenAI's Preparedness Framework. If a benchmark itself is flawed, it can give a false picture of model capabilities, misrepresent safety cases, and skew research priorities.

SWE-Bench Pro programmatically sources tasks from the history of feature changes in public and private repositories. On its 731-task public split, frontier models' pass rate jumped from 23.3% to 80.3% in just eight months. If a benchmark that the whole community leans on turns out to be standing on shaky ground, that's a big deal — which is exactly why this audit matters.

What changes

In light of these findings, OpenAI has retracted its earlier recommendation to adopt SWE-Bench Pro, and is advising model developers to carefully examine results from it rather than take the numbers at face value.

OpenAI is also hoping the wider evaluation community will build new benchmarks designed from the start by experienced software developers specifically to test model capabilities.

Dive Deep

The issues fell into four main categories.

• Overly strict tests — enforce specific implementation details not stated in the prompt, invalidating functionally correct submissions
• Underspecified prompts — omit requirements that hidden tests enforce but that aren't reasonably inferable
• Low-coverage tests — don't adequately check the requested feature, letting incomplete fixes pass
• Misleading prompts — point models toward the wrong behavior or contradict what the tests actually require

The audit process itself was pretty thorough. An initial automated filter reviewed the instructions given to the model, the model's attempts, and the grading tests, flagging 286 potentially broken tasks. From there, OpenAI dug deeper along two paths.

The first was a human-supervised agent review: Codex-based investigator agents with access to the task repository and environment ran tests and inspected files to distinguish reasonable ambiguity from true underspecification, with a researcher making the final call after multiple independent passes.

The second was a human annotation campaign, where five experienced software engineers — trained on the benchmark's goals and issue taxonomy — independently reviewed the problem statement, test cases, and gold patch before forming a judgment.

Interestingly, human reviewers were more likely than the agents to label a task as broken. The two approaches' flagged categories overlapped in 74% of cases, and the biggest gap was around low-coverage tests: humans selected it as the top issue for 9.4% of the benchmark, versus only 4.1% from the agent pipeline.

OpenAI points to a root cause: issues and pull requests from open-source repositories were originally written for human collaboration, not to form clean, isolated evaluation tasks. Tests written to validate one specific change often end up overly strict rather than defining an implementation-agnostic standard. On the flip side, more capable models now make it easier to inspect prompts, tests, patches, and failure traces at scale, which is exactly what made this kind of audit possible.

Wrap-up

• OpenAI audited SWE-Bench Pro and found roughly 30% of tasks broken — 27.4% flagged by an automated pipeline, 34.1% by human reviewers
• The problems fall into four categories: overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts
• OpenAI has retracted its earlier recommendation to adopt SWE-Bench Pro and urges model developers to scrutinize results carefully
• The audit combined Codex-based investigator agents with a five-person human annotation team

If you follow AI model benchmark results, this is a good reminder to think twice before taking the numbers at face value.