Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.
Open source
24 alternativesHPOlib2
HPOBench is a collection of benchmark problems for hyperparameter optimization.
About HPOlib2
HPOlib2 (hosted as the HPOBench repository) is an open-source Python library that provides benchmark problems for hyperparameter optimization, including support for multi-fidelity optimization workflows, with a focus on reproducibility. It offers pre-built containerized benchmarks using Singularity 3.6 to isolate dependencies and ensure consistent results over time, as well as local benchmark options that require manual dependency installation which may cause conflicts with other software.
Key features
- Supports multi-fidelity hyperparameter optimization benchmarks
- Offers containerized benchmarks for dependency isolation and reproducibility
- Includes local benchmark options for users without Singularity
- Allows querying benchmarks with adjustable or no fidelity parameters
- Provides access to configuration, fidelity space and benchmark metadata
- Supports building custom local benchmark containers with Singularity
Best for: Researchers and developers building or evaluating hyperparameter optimization algorithms who need standardized, reproducible benchmark tasks.
Good for
- Containerized benchmarks avoid external dependency changes for consistent results
- Supports both multi-fidelity and standard hyperparameter optimization workflows
- Includes simple code examples for quick benchmark evaluation
Watch out for
- Local benchmark runs may require complex manual dependency installation
- Containerized benchmarks require Singularity version 3.6 to operate
- Singularity is not included in the standard pip installation
HPOlib2 at a glance
- Website
- github.com/automl/HPOBench
- Category
- AI & Machine Learning › AI Data & Analytics
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
HPOlib2 alternatives
Tools and sites you could use instead, ranked by how closely they match.
WhichLLM tests local language models on a user’s hardware and ranks them using recent benchmarks.
Open source
24 alternatives
Giskard provides open source evaluation and testing tools for LLM agents.
Open source
24 alternatives
NALP covers Natural Adversarial Language Processing through a dedicated library.
Open source
24 alternatives
neuropredict evaluates predictive performance using neuroimaging features.
Open source
24 alternatives
Adala automates data labeling through an autonomous agent framework for organizing labeled datasets.
Open source
24 alternativesPopular in AI Data & Analytics
Well-known listings from the same topic.
Dotgetdot.aiForvioforvio.comFluidAI Medicalfluidai.mdApache Zeppelinzeppelin.apache.orgSola Securitysola.securityQuadraticquadratichq.com
HPOlib2 FAQ
Do I need Singularity to use HPOlib2?
No, local benchmark options are available, though they require manual dependency installation that may conflict with other software.
Can I adjust fidelity levels when running a benchmark?
Yes, you can specify custom fidelity values or omit fidelity to run at the highest default level.
Where is HPOlib2 configuration data stored by default?
By default, it is stored in ~/.config/hpobench, or the path set by the $XDG_CONFIG_HOME environment variable.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.