Definition
SWE-bench is SWE-bench is a coding benchmark that asks an AI system to fix real software bugs taken from GitHub issues, and counts a fix as correct only if the project's own tests then pass.
SWE-bench is a test for AI coding systems built from real work. Its full set has 2,294 tasks drawn from issues and pull requests in 12 popular open-source Python projects. For each task the AI gets the project's code and the bug report, then has to write a patch.
The grading is mechanical. After the AI's patch is applied, the project's own tests run. If the tests that failed before now pass, and the tests that passed before still pass, the task counts as solved. A score is the percentage of tasks solved. No human has to judge the answer.
That makes it more realistic than a quiz of small puzzles. The AI has to find the right files in a large codebase, understand the problem and change the code without breaking anything else.
Some original tasks were flawed: vague descriptions, or tests that rejected valid fixes. In August 2024, OpenAI worked with the benchmark's authors to release SWE-bench Verified, a 500-task subset that human annotators judged clear and solvable. It is the version most labs quote when they report a "SWE-bench" score.
If you choose a model for coding work, SWE-bench is a useful first filter, because it measures finishing real tasks. It is not a substitute for trying the model on your own codebase, where your conventions, tests and languages differ.
It is a 500-task subset of SWE-bench that human annotators checked for clear descriptions and fair tests, released by OpenAI in August 2024.
No. It measures fixing bugs in a set of open-source Python projects with a particular setup, so test a model on your own code too.
Anthropic says new defenses cut the Claude reasoning data distillers harvest by about 90 to 99%, and it has banned tens of thousands of accounts. The fine print reaches honest developers too: new API accounts can't edit Claude's prior context, and blocked requests are billed.