TensorRT-LLM provides Python and C++ runtimes with optimizations for language model inference on NVIDIA GPUs.
FastChat
FastChat offers an open platform for training, serving, and evaluating large language models, and hosts release repositories for Vicuna and Chatbot Arena.
About FastChat
FastChat is an open-source platform for training, serving and evaluating large language model chatbots. It provides the training and evaluation code behind models such as Vicuna and MT-Bench, plus a distributed multi-model serving system with a web UI and OpenAI-compatible RESTful APIs. The project also powers Chatbot Arena (lmarena.ai), which has served over 10 million chat requests for 70+ LLMs and collected over 1.5 million human votes for an online LLM Elo leaderboard.
It can be installed with pip or built from source and run from the command line, with model weights for Vicuna, LongChat and FastChat-T5 downloaded automatically from Hugging Face repositories into a local cache folder. Vicuna is based on Llama 2 and is intended to be used under Llama's model license. Some context lengths and CLI output modes carry extra requirements: 16K Vicuna versions need transformers >= 4.31, and the rich-text CLI style is described as experimental and may not work on certain terminals. No pricing information is published for the repository.
Key features
- Training and evaluation code for models such as Vicuna and MT-Bench
- Distributed multi-model serving system with web UI
- OpenAI-compatible RESTful APIs for served models
- Command-line inference with automatic Hugging Face weight downloads
- Installation via pip or from source, with Docker files included
- Supports many model families including Llama 2, Falcon, and ChatGLM
Best for: Developers and researchers who need open-source tooling to train, serve and evaluate large language model chatbots.
Good for
- Open-source platform covering training, serving and evaluation of LLM chatbots
- Underlies Chatbot Arena, which has served over 10 million chat requests for 70+ LLMs
- Supports a wide range of model families and provides OpenAI-compatible APIs
Watch out for
- Vicuna weights are based on Llama 2 and should be used under Llama's model license
- 16K context versions require transformers >= 4.31
- Rich-text CLI output is experimental and may not work on some terminals
FastChat at a glance
- Website
- github.com/lm-sys/FastChat
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 5, 2026 · Online when checked
- Listed
- Oct 5, 2026
FastChat alternatives
Tools and sites you could use instead, ranked by how closely they match.
Sagify describes its project as a way to work with large language models and machine learning more easily.
Open source
24 alternatives
DeepLake provides multimodal data infrastructure for AI agents, including retrieval and training.
Open source
24 alternatives
Machine learning operations and large language model operations are the focus of Starwhale’s platform.
Open source
24 alternatives
OptiMate is a collection of libraries for optimizing AI model performance; the repository is marked as a legacy project.
Open source
24 alternatives
traceAI provides open source tracing for AI applications and frameworks using OpenTelemetry.
Open source
24 alternativesPopular in AI Infrastructure & MLOps
Well-known listings from the same topic.
Guild AIguild.aiFlow AIflow-ai.comsystemprompt.iosystemprompt.ioKili Technologykili-technology.comAPI docsdocs.x.aiEyePop.aieyepop.ai
FastChat FAQ
What is FastChat used for?
It is an open platform for training, serving and evaluating large language model based chatbots, and includes the training and evaluation code for models such as Vicuna and MT-Bench.
Do I need to download model weights separately?
The provided commands start chatting and automatically download weights from Hugging Face repositories, which are stored in a cache folder in the user's home directory.
What license applies to Vicuna?
Vicuna is based on Llama 2 and should be used under Llama's model license.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.