Looking for something like TensorRT-LLM? Here are 24 alternatives, ranked by how closely they match it on purpose, features and platforms. Of these, 1 are free or have a free plan and 18 are open source.
TensorRT-LLM provides Python and C++ runtimes with optimizations for language model inference on NVIDIA GPUs.
| # | Alternative | Pricing | Open source | Platforms |
|---|---|---|---|---|
| 1 | FastChat | — | Yes | — |
| 2 | deeplake | — | Yes | — |
| 3 | Forefront | Freemium | Not stated | Web |
| 4 | VDP | — | Not stated | — |
| 5 | Sagify | — | Yes | — |
| 6 | QuivrHQ/MegaParse | — | Yes | — |
| 7 | MiroShark | — | Yes | — |
| 8 | Upstage | — | Not stated | Web |
| 9 | Nebullvm | — | Yes | — |
| 10 | Ray | — | Yes | — |
FastChat offers an open platform for training, serving, and evaluating large language models, and hosts release repositories for Vicuna and Chatbot Arena.
DeepLake provides multimodal data infrastructure for AI agents, including retrieval and training.
Forefront provides a platform for fine tuning and running inference with open source language models.
Instill Core coordinates data, models, and pipelines for AI first applications.
Sagify describes its project as a way to work with large language models and machine learning more easily.
File Parser optimised for LLM Ingestion with no loss. Parse PDFs, Docx, PPTx in a format that is ideal for LLMs.
MiroShark describes a swarm intelligence engine for simulating scenarios with stated time and cost targets.
Upstage builds large language models and document-processing engines for workplace workflows.
OptiMate is a collection of libraries for optimizing AI model performance; the repository is marked as a legacy project.
Ray provides an AI compute engine with a distributed runtime and libraries for machine learning workloads.
traceAI provides open source tracing for AI applications and frameworks using OpenTelemetry.
Helicone provides open-source observability for LLM applications, with tools to monitor, evaluate, and experiment.
Hyperopt supports distributed asynchronous hyperparameter optimization in Python.
DeepDetect provides a deep learning server and command-line interface for Torch and TensorRT.
OneCompression is a Python package for compressing large language models.
Machine learning operations and large language model operations are the focus of Starwhale’s platform.
GPT-NeoX implements model parallel autoregressive transformers on GPUs using Megatron and DeepSpeed libraries.
dstack coordinates AI compute and supports training and inference across GPU clouds, Kubernetes, VMs, and bare metal.
BLACKBOX AI serves selected open-weight models as dedicated deployments or through one inference endpoint.
OpenLLMetry provides OpenTelemetry-based observability for generative AI and large language model applications.
GPTCache provides semantic caching for large language models and integrates with LangChain and LlamaIndex.
Mosec serves machine learning models with dynamic batching and CPU or GPU pipelines for inference workloads.
BentoML helps build and serve AI apps and models through inference APIs, job queues, and multi-model pipelines.
Extensible Toolkit for Finetuning and Inference of Large Foundation Models.
Alternatives are suggested from the purpose and features described in our listings. Different pages for the same product appear once. Read each description to check whether it fits your task. How we choose alternatives · Suggest an alternative
