SWE-bench is the benchmark that measures whether language models and coding agents can resolve real-world GitHub issues. Given a repository snapshot and an issue description, the system under test must produce a patch that makes the project’s own test suite pass. The project has about 6,100 stars on GitHub, is MIT licensed, and was actively maintained in 2026.
The original dataset contains 2,294 task instances drawn from real pull requests in 12 popular Python repositories. Evaluation is execution-based: each generated patch is applied in a Dockerized environment and graded on FAIL_TO_PASS tests (previously failing tests that should now pass) and PASS_TO_PASS tests (tests that must not break). The family has grown to include SWE-bench Verified, a human-validated subset maintained with OpenAI, a smaller Lite split for cheaper runs, and a Multimodal variant. Results from labs and agent builders are published on the official leaderboard at swebench.com, making it the standard yardstick for software-engineering agents.
The benchmark’s influence is hard to overstate: when a new frontier model or coding agent is announced, its SWE-bench Verified score is usually the headline number, and the harness’s strict, test-based grading is a big reason those results are trusted. SWE-bench has also spawned a research ecosystem of training datasets, such as SWE-smith and SWE-Gym, that reuse the same task format to teach models โ extending the project’s impact well beyond evaluation alone.
Free. SWE-bench is open source under the MIT license; you only pay for the compute and model API calls your evaluation consumes.
AI labs, agent builders and researchers who need a credible, reproducible way to measure how well a model or agent performs on real software-engineering work.