Prompt Engineering Guide collects lessons, papers, notebooks, and resources on prompting, retrieval-augmented generation, and AI agents.
Open source
24 alternativesPromptfoo
promptfoo tests prompts, agents, and retrieval-augmented generation systems, with red teaming and vulnerability scanning.
About Promptfoo
Promptfoo is a command-line tool and library for evaluating and red-teaming applications built on large language models (LLMs). It is used to test prompts, agents, and retrieval-augmented generation (RAG) systems with automated evaluations, and to compare how models such as OpenAI, Anthropic, Azure, Bedrock, and Ollama models perform on the same tests. Configurations are written as declarative files and can run in CI/CD pipelines.
It also includes red teaming and vulnerability scanning for LLM apps, a code scanning option for reviewing pull requests for LLM-related security and compliance issues, and report generation for security vulnerabilities. The source states that LLM evaluations run locally on the user's machine, but most LLM providers require an API key, and model calls go to those external services. The project is MIT licensed and open source. The source says it is now part of OpenAI. Pricing is not stated.
Key features
- Automated evaluations of prompts, agents, and RAG systems
- Red teaming and vulnerability scanning for LLM applications
- Side-by-side model comparison across multiple LLM providers
- Declarative configuration files run from the command line
- CI/CD integration for automated checks
- Code scanning of pull requests for LLM-related security issues
Best for: Developers and engineering teams building LLM-based applications who want to test prompts, compare models, and scan for security issues from the command line or in CI/CD pipelines.
Good for
- Open source and MIT licensed
- The source states that LLM evaluations run locally on the user's machine
- Supports comparison across several LLM providers, including OpenAI, Anthropic, Azure, Bedrock, and Ollama
Watch out for
- Most LLM providers require an API key, which must be set as an environment variable
- Model calls to external providers are handled by those services, outside the local evaluation process
Promptfoo at a glance
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
Promptfoo alternatives
Tools and sites you could use instead, ranked by how closely they match.
This repository curates AI development resources, prompts, frameworks, and configuration examples.
Open source
24 alternatives
A curated list of generative AI tools, projects, models, and references.
Open source
24 alternatives
This repository curates prompts for Google Gemini image generation, with previews and multiple language versions.
Run the llm command-line tool to access and work with large language models from a terminal.
Open source
24 alternatives
This repository teaches 22 prompt engineering techniques through Jupyter Notebook tutorials.
Popular in AI Directories & Resources
Well-known listings from the same topic.
getdesign.mdgetdesign.mdAI Valleyaivalley.aifuturepediafuturepedia.ioAwesomeclaudeawesomeclaude.aiLLM Explorerllm-explorer.comAI Depotaidepot.co
Promptfoo FAQ
How can Promptfoo be installed?
The source lists npm install -g promptfoo, brew install promptfoo, and pip install promptfoo. It can also be run without installing by using npx promptfoo@latest.
Do I need an API key to use it?
Most LLM providers require an API key. The source says to set it as an environment variable, for example export OPENAI_API_KEY=sk-abc123.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.