Written and maintained by CASRAI Editorial Board
Last updated
SWE-bench is a benchmark that scores AI systems on whether they can fix real GitHub issues: a task counts as solved only if the project’s own tests pass after the model’s patch is applied. This guide is for researchers, evaluators and governance readers who have to read, cite or compare SWE-bench numbers. Before you quote a score, pin down the split, the agent setup, who ran it, and whether the tests are a reliable judge.
What it is
SWE-bench is an evaluation framework introduced in the paper “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (Jimenez et al., ICLR 2024). SWE stands for software engineering. Given a codebase and the text of an issue, a language model must edit the codebase to address that issue. The original set holds 2,294 task instances drawn from 12 popular Python repositories.
How resolution is scored
Each task instance is built from a merged pull request that resolves an issue and changes the repository’s test files. Every instance has at least one FAIL_TO_PASS test, a test whose status flips from fail to pass once the reference fix is applied, and 40% of instances have at least two. A median of 51 further tests (PASS_TO_PASS) check that existing behaviour still works. The model’s patch is applied inside a Docker container and the tests are run; an instance counts as resolved only if every FAIL_TO_PASS and PASS_TO_PASS test passes. The leaderboard figure “% Resolved” is the percentage of instances solved.
Who runs it and what it covers
The code lives in the princeton-nlp GitHub organisation, and the swebench.com leaderboards are credited to the SWE-bench Team, which thanks Open Philanthropy, AWS, Modal, Andreessen Horowitz, OpenAI and Anthropic. The official site lists these splits:
| Split | Instances | What it is |
|---|---|---|
| SWE-bench (full) | 2,294 | The original benchmark: real GitHub issues from 12 Python repositories |
| Lite | 300 | A subset curated for less costly evaluation |
| Verified | 500 | A human-filtered subset of SWE-bench |
| Multilingual | 300 | Tasks from 42 repositories across 9 programming languages |
| Multimodal | 480 | Issues described with visual elements |
SWE-bench Verified
Verified is a human-filtered subset of 500 instances, created in collaboration with OpenAI. Annotators reviewed each instance to confirm that the problem description is clear, the test patch is correct and the task is solvable. It was released on 13 August 2024.
The dataset’s own difficulty field shows what the subset contains: 194 instances are labelled as a fix of under 15 minutes, 261 as 15 minutes to 1 hour, 42 as 1-4 hours and 3 as over 4 hours, so 455 of 500 sit under an hour. It is also concentrated in a few repositories:
| Repository | Full (2,294) | Verified (500) |
|---|---|---|
| Django | 850 (37%) | 231 (46%) |
| SymPy | 386 | 75 |
| scikit-learn | 229 | 32 |
| Sphinx | 187 | 44 |
| matplotlib | 184 | 34 |
| Seven other repositories combined | 458 | 84 |
Django and SymPy together supply 54% of the full set and 61% of Verified, so a Verified score is largely a statement about those two codebases.
SWE-bench Pro
SWE-Bench Pro (Scale AI authors, arXiv September 2025) is a separate, harder benchmark built on SWE-bench’s practices. It has 1,865 problems from 41 repositories, split into a public set of 731 instances from 11 GPL-licensed repositories (released on Hugging Face), a held-out set of 858 from 12 repositories, and a commercial set of 276 from 18 proprietary startup repositories that stays private, with results reported in the paper. Reference solutions average 107.4 lines across 4.1 files. The paper reports Pass@1 as the resolve rate and found performance below 45% under a unified scaffold.
How to search or use it
These steps follow the official swebench.com site and documentation.
Read the leaderboards
- Open swebench.com and choose a leaderboard: Verified, Multimodal, Multilingual, Lite or Full.
- On Verified, pick the view. “Bash Only” runs every model in the same mini-SWE-agent environment; the Agent dropdown switches to the full leaderboard of all agents.
- Read the “% Resolved” column and look for the marker for runs performed or directly checked by the SWE-bench team.
- Tick models in the first columns, or use Quick select (top 10, top 20, all, all open-weights), then click “Compare results”.
Run an evaluation yourself
- Clone github.com/princeton-nlp/SWE-bench, run
pip install -e ., and make sure Docker is installed and running. - Load a split from Hugging Face, for example
load_dataset('princeton-nlp/SWE-bench_Verified', split='test'). - Write predictions as JSONL with the fields
instance_id,model_name_or_pathandmodel_patch. - Run
python -m swebench.harness.run_evaluation --dataset_name princeton-nlp/SWE-bench_Lite --predictions_path predictions.jsonl --max_workers 8 --run_id my_run. Results are written underlogs/evaluation/my_run/, with a summary inresults.jsonand areport.jsonper instance. To check your setup first, run the ground-truth (“gold”) patches.
The harness caches results by run_id and instance, so re-evaluating a changed patch needs a new run_id.
Strengths and limits
Strengths
- Real tasks, executable judging. Issues and tests come from real repositories, and scoring runs the repository’s tests in a containerised, repeatable harness instead of comparing text.
- Regression guard. A patch that fixes the issue but breaks other behaviour fails the PASS_TO_PASS tests.
Limits
- Tests are the oracle, and they can be weak. UTBoost (ACL 2025) found 36 task instances with insufficient test cases and 345 erroneous patches incorrectly labelled as passed. The corrections affect 40.9% of Lite and 24.4% of Verified leaderboard entries and change 18 and 11 rankings respectively. The Pro authors flag the same dependence on the test suite in their own benchmark, since valid solutions may fail tests that expect specific behaviour.
- Public repositories invite contamination. The Pro authors call contamination a weakness of many coding benchmarks and use GPL-licensed and private codebases as their defence. For the general concept see data leakage in training.
- A score is model plus scaffold. The Verified leaderboard mixes systems ranging from simple language-model agent loops to retrieval systems and multi-rollout review systems, which is why the site offers the Bash Only view for like-for-like model comparison. A study of 79 Lite and 133 Verified entries (Martinez and Franch, ICSE-SEIP 2026) found most come from industry, with proprietary models dominant.
- Splits are not interchangeable. The original paper’s best model, Claude 2 with BM25 retrieval, resolved 1.96% of the full set; the site’s news records mini-SWE-agent at 65% on Verified in July 2025. Different split and setup: the gap shows fast movement, not one trend line.
A SWE-bench number is a capability measurement. It is usually one component of a model evaluation suite, and the access and independence questions around who runs such tests are covered in third-party evaluator standards and methodology. Forecasting studies have also used it as a target; see emergent abilities and capability forecasting.
Alternatives
| Benchmark | What it tests | How it differs from SWE-bench |
|---|---|---|
| SWE-Bench Pro | 1,865 long-horizon problems from 41 repositories | Multi-file tasks; public, held-out and commercial sets |
| HumanEval | Self-contained coding problems | The SWE-bench paper contrasts it with repository-level work: problems solvable in a few lines of code |
| ProgramBench | Whether models can code meaningful software artifacts from scratch | Released by the SWE-bench team in May 2026; builds from nothing rather than patching a repository |
Outside coding, Humanity’s Last Exam tests closed-ended expert academic questions and the MMLU benchmark tests broad academic knowledge.
Frequently asked questions
What is SWE-bench Verified benchmark?
SWE-bench Verified is a 500-instance, human-filtered subset of SWE-bench created with OpenAI. Annotators checked that each problem description is clear, the test patch is correct and the task is solvable. Scoring is unchanged: percentage of instances resolved. 455 of its 500 instances carry a difficulty label under one hour.
What is the best AI performance benchmark?
No single benchmark is best, because each measures a different capability. SWE-bench measures repository-level issue fixing in Python, Humanity’s Last Exam measures expert academic questions, and MMLU measures broad knowledge. Match the benchmark to the capability you need to evidence, then check the split, agent setup and test quality described above.
What does SWE stand for?
SWE stands for software engineering. The SWE-bench paper frames its tasks as real-world software engineering problems taken from GitHub issues.
What is a benchmark in coding?
A coding benchmark is a fixed set of programming tasks with an automatic scoring rule, usually unit tests that a generated solution must pass. HumanEval-style benchmarks use self-contained problems; SWE-bench instead asks for a patch to a real repository and scores it with that repository’s own tests.
Sources
Official pages and papers, as of 7 November 2026.
- Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K. R. “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” ICLR 2024; arXiv:2310.06770v3 (11 November 2024): https://arxiv.org/abs/2310.06770.
- SWE-bench Team. SWE-bench leaderboards and Verified page: https://www.swebench.com/ and https://www.swebench.com/verified.html.
- SWE-bench documentation, Quickstart and Evaluation guide: https://www.swebench.com/SWE-bench/guides/quickstart/ and https://www.swebench.com/SWE-bench/guides/evaluation/.
- Hugging Face. Dataset card princeton-nlp/SWE-bench_Verified (difficulty and repository counts from the dataset viewer): https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified.
- Deng, X., Da, J., et al. “SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?” arXiv:2509.16941v2 (14 November 2025): https://arxiv.org/abs/2509.16941.
- Yu, B., Zhu, Y., He, P., Kang, D. “UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench.” ACL 2025; arXiv:2506.09289: https://arxiv.org/abs/2506.09289.
- Martinez, M., Franch, X. “What’s in a Benchmark? The Case of SWE-Bench in Automated Program Repair.” ICSE-SEIP 2026; arXiv:2602.04449: https://arxiv.org/abs/2602.04449.
