FastChat offers an open platform for training, serving, and evaluating large language models, and hosts release repositories for Vicuna and Chatbot Arena.
Open source
24 alternativesTensorRT-LLM
TensorRT-LLM provides Python and C++ runtimes with optimizations for language model inference on NVIDIA GPUs.
About TensorRT-LLM
TensorRT-LLM is a repository of software for running inference on large language models (LLMs) on NVIDIA GPUs. It provides a Python API for defining LLMs and includes components for building Python and C++ runtimes that manage how inference runs.
The project describes its code as optimizing inference with specialized kernels for common operations, an efficient runtime, and a Python-based framework that users can customize and extend. Its overview also states that it supports Visual Gen models. The repository's blog posts discuss topics such as speculative decoding, disaggregated serving, and expert parallelism. The source describes the software as targeting NVIDIA GPUs.
Key features
- Python API for defining large language models
- Specialized kernels for common model operations
- Python and C++ runtimes that orchestrate inference execution
- Pythonic framework for customizing and extending the system
- Support for Visual Gen models, according to the repository overview
- Technical posts cover speculative decoding, disaggregated serving, and expert parallelism
Best for: Developers and engineers who run LLM inference on NVIDIA GPUs and want to customize the serving software.
Good for
- Provides a Python API for defining LLMs
- Includes Python and C++ runtimes for managing inference execution
- Offers a framework that users can customize and extend
Watch out for
- Targets NVIDIA GPUs; the source describes no support for other hardware
TensorRT-LLM at a glance
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Not stated
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
TensorRT-LLM alternatives
Tools and sites you could use instead, ranked by how closely they match.
DeepLake provides multimodal data infrastructure for AI agents, including retrieval and training.
Open source
24 alternatives
Forefront provides a platform for fine tuning and running inference with open source language models.
Freemium
24 alternatives
Instill Core coordinates data, models, and pipelines for AI first applications.
Sagify describes its project as a way to work with large language models and machine learning more easily.
Open source
24 alternatives
File Parser optimised for LLM Ingestion with no loss. Parse PDFs, Docx, PPTx in a format that is ideal for LLMs.
Open source
24 alternativesPopular in AI Infrastructure & MLOps
Well-known listings from the same topic.
Guild AIguild.aiFlow AIflow-ai.comsystemprompt.iosystemprompt.ioKili Technologykili-technology.comAPI docsdocs.x.aiEyePop.aieyepop.ai
TensorRT-LLM FAQ
What hardware does TensorRT-LLM target?
The source describes TensorRT-LLM as providing inference optimizations for language models on NVIDIA GPUs.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.