This repository curates research on large language models for code generation.
Open source
24 alternativesSwe Bench
SWE-bench evaluates whether language models can resolve real GitHub issues.
About Swe Bench
SWE-bench is an open-source benchmark for evaluating large language models on real-world software issues collected from GitHub. Given a codebase and an issue, a model is tasked with generating a patch that resolves the problem.
Datasets are accessed through Hugging Face and evaluated with a Docker-based harness and command-line tools. The v5 CLI builds evaluation images from a task repository, and setup requires installing Docker and installing the package from source.
Key features
- Evaluate model-generated patches against real GitHub issues
- Load benchmark splits from Hugging Face with Python
- Run containerized evaluations using Docker
- Evaluate verified, multimodal, multilingual, and full datasets
- Generate predictions with mini-SWE-agent via CLI
- Publish evaluation runs to a Hugging Face bucket
Best for: Researchers evaluating language models on resolving real GitHub software issues.
Good for
- Open source code and data for benchmark tasks
- Uses Docker containers for reproducible evaluations
- Provides multiple dataset variants including verified and multimodal
Watch out for
- Requires Docker installation and setup to run evaluations
- v5 CLI requires building images from a task repository
- ARM-based systems such as M-series Macs need local builds with --task-repo
Swe Bench at a glance
- Category
- AI & Machine Learning › Models & ML Research
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 5, 2026 · Online when checked
- Listed
- Oct 5, 2026
Swe Bench alternatives
Tools and sites you could use instead, ranked by how closely they match.
This repository lists open language models identified as available for commercial use.
Open source
24 alternatives
Microsoft’s loralib implements low rank adaptation for large language models.
Open source
24 alternatives
DeepSeek-VL2 supports multimodal understanding through a mixture-of-experts vision-language model.
Open source
24 alternatives
Qwen is a series of chat and pretrained language models proposed by Alibaba Cloud.
Open source
24 alternatives
Stability AI’s StableLM repository contains its language models and related project materials.
Open source
24 alternativesPopular in Models & ML Research
Well-known listings from the same topic.
Falcon LLMfalconllm.tii.aeERNIEernie.baidu.comTokenVersetoken-verse.github.iowTAObittensor.comAlpacacrfm.stanford.edu/2023/03/13/alpaca.htmlSmolVLAhuggingface.co/blog/smolvla
Swe Bench FAQ
How do I load the SWE-bench dataset?
Use Python with from datasets import load_dataset, then load_dataset('princeton-nlp/SWE-bench', split='test').
How do I evaluate patch predictions?
Run swebench eval verified -p --run-id -j .
What is needed before running SWE-bench?
Install Docker following the setup guide, then clone the repository and run pip install -e .
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.