OpenLM Arena compares open language models through leaderboards and benchmark results.
qcri/LLMeBench
LLMeBench provides a flexible framework for benchmarking large language models.
About qcri/LLMeBench
LLMeBench is a Python framework for benchmarking large language models (LLMs) on natural language processing (NLP) tasks. It runs evaluations using assets, which are files that specify a dataset, a model, and a task. The repository includes a set of benchmarking assets that can be downloaded, and users can create their own assets for new tasks, models, or datasets in any language.
The framework supports several model providers, including OpenAI, the Hugging Face Inference API, Azure, and Petals. Some datasets are included with automatic download, but the dataset files for some datasets cannot be redistributed. Users must obtain those from their original sources. The dataset disclaimer states that the licenses of the included datasets are understood to allow use for research or non-commercial purposes.
Key features
- Evaluates NLP tasks across OpenAI, Hugging Face Inference API, Azure, and Petals models
- Includes 34 tasks, 7 model providers, and about 800 ready-to-run benchmarking assets
- Supports text, speech, and multimodal evaluation
- Caches API results to avoid repeated calls in repeated experiments
- Supports zero-shot and few-shot evaluation settings
- Downloads datasets on the fly and caches them locally
Best for: Researchers and developers who want to benchmark LLMs on NLP tasks through a command-line Python workflow.
Good for
- Covers 34 tasks, 80 datasets, and 16 languages in its tested asset set
- Caching reduces repeated API calls during repeated experiments
- Automatically downloads datasets when possible and provides ready-to-run assets
Watch out for
- Some dataset files cannot be redistributed and must be obtained from original sources
- Dataset licenses are described as allowing research or non-commercial use
- Setup and runs are done from the command line using pip and Python
qcri/LLMeBench at a glance
- Website
- github.com/qcri/LLMeBench
- Category
- AI & Machine Learning › AI Data & Analytics
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Not stated
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
qcri/LLMeBench alternatives
Tools and sites you could use instead, ranked by how closely they match.
FastDatasets helps create training datasets for large language models.
Open source
24 alternatives
Automatic "Differentiation" via Text, using large language models to backpropagate textual gradients.
Open source
24 alternatives
Uncover LLM evaluation's importance and explore methods for assessing its performance and impact across industries.
WhichLLM tests local language models on a user’s hardware and ranks them using recent benchmarks.
Open source
24 alternatives
Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Open source
24 alternativesPopular in AI Data & Analytics
Well-known listings from the same topic.
Dotgetdot.aiForvioforvio.comFluidAI Medicalfluidai.mdApache Zeppelinzeppelin.apache.orgSola Securitysola.securityQuadraticquadratichq.com
qcri/LLMeBench FAQ
Which model providers does LLMeBench support?
The README lists OpenAI (for example GPT), the Hugging Face Inference API, Azure, and Petals (for example BLOOMZ). It also states that the overview lists 7 model providers in total.
Do I have to download datasets manually?
No, for many datasets the framework downloads them automatically when possible. Manual download is also available, and some datasets must be acquired from their original sources by the user.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.