SWE-bench Verified maxed out, and it's time to build your own private coding evals

OpenAI has officially stopped caring about SWE-bench Verified, and you probably should too. Their explanation for abandoning the benchmark reveals that the metric has become contaminated and is no longer suitable for measuring frontier coding progress. State-of-the-art progress slowed to a crawl, and OpenAI found that in a subset of frequently failed tasks, nearly 60% had flawed test cases that rejected functionally correct solutions.

This isn't just inside-baseball drama; it is a blaring alarm that evaluation confidence expires. SWE-bench did its job in making coding-agent progress measurable, but benchmarks are instruments, and instruments drift. As models train on more public repositories, tests based on those repositories lose their signal. A public leaderboard might tell you which model is trending, but it cannot tell you if an AI agent will survive your team's specific flavor of legacy code, bespoke testing frameworks, and undocumented dependencies.

Builders need to stop letting yesterday's leaderboard dictate today's production decisions and start building private, adversarial evaluations based on their actual work. This means testing on boring code, old code, and weird dependency upgrades, tracking not just pass rates but human review time, requested changes, and defects after merge. An AI that can write syntax but fails at task selection, test integration, and rollback management is just an intern with a faster typing speed. SWE-bench hitting a ceiling is a sign that engineering teams need to grow up and realize that maintaining private evaluations is now a permanent part of the job.