NobodyWho runs language models locally through an inference engine designed for efficient operation across devices.
Open source
22 alternativesOpenLLM
OpenLLM runs open-source language models as cloud API endpoints compatible with OpenAI clients.
About OpenLLM
OpenLLM is an open-source tool from BentoML for running open-source language models as OpenAI-compatible API endpoints. After installing it with pip, a server is started with a single command such as "openllm serve llama3.2:1b"; by default the API is available at http://localhost:3000 and can be called by clients and frameworks that support OpenAI-compatible APIs, including the OpenAI Python client and LlamaIndex. It also provides a chat UI at the /chat endpoint and CLI chat through "openllm run".
Supported models listed by the project include DeepSeek R1, Gemma, Jamba, Llama 3.1 through Llama 4, Mistral, Phi, Pixtral, Qwen2.5 and QwQ, and a model repository can be added to run custom models. GPU requirements differ per model, ranging from roughly 12G for small models to 80G x 16 for DeepSeek r1-671b. OpenLLM does not store model weights, and a Hugging Face token is required for gated models. Deployment workflows are described for Docker, Kubernetes and BentoCloud. Pricing information is not stated.
Key features
- Serves open-source LLMs as OpenAI-compatible API endpoints with one command
- Built-in chat UI at /chat and CLI chat via openllm run
- Supports Llama, Qwen, Gemma, Mistral, Phi, DeepSeek and Pixtral models
- Allows adding a model repository to run custom models
- Deployment workflows for Docker, Kubernetes and BentoCloud
- API key for client authentication is optional by default
Best for: Developers and teams who want to self-host open-source LLMs behind OpenAI-compatible APIs on their own machines or cloud infrastructure.
Good for
- Open source and installable with pip, with a server started by a single command
- OpenAI-compatible endpoints work with existing OpenAI clients and tools such as LlamaIndex
- Documented deployment paths for Docker, Kubernetes and BentoCloud
Watch out for
- Model weights are not stored, and gated models require a Hugging Face token and access approval
- GPU requirements vary by model and can be very large, up to 80G x 16 for DeepSeek r1-671b
OpenLLM at a glance
- Website
- github.com/bentoml/OpenLLM
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
OpenLLM alternatives
Tools and sites you could use instead, ranked by how closely they match.
GPUStack manages GPU clusters for serving AI models and provides on demand GPU instances accessible through SSH.
Open source
21 alternatives
For Apple Silicon, Rapid-MLX offers an open-source LLM inference server and Mac app compatible with OpenAI and Anthropic APIs.
LLMKube runs self hosted LLM inference across heterogeneous GPU fleets using Kubernetes and several model runtimes.
Open source
22 alternatives
PowerInfer supports serving large language models for local deployment and inference.
Open source
23 alternatives
bitsandbytes enables large language models in PyTorch using k-bit quantization.
Open source
21 alternativesPopular in Model Serving & Inference
Well-known listings from the same topic.
MAI Playgroundplayground.microsoft.ai/chatBigModelbigmodel.cnkimi-k2.6platform.kimi.ai/docs/pricing/chat-k26Jinagithub.com/jina-ai/jinaFluxgithub.com/black-forest-labs/fluxprivateGPTgithub.com/imartinez/privateGPT
OpenLLM FAQ
Do I need a Hugging Face token to serve a model?
Yes for gated models. OpenLLM does not store model weights, so an HF_TOKEN environment variable must be set and access requested for the gated model.
Where is the API available after starting a server?
By default the server is accessible at http://localhost:3000, with OpenAI-compatible APIs under /v1 and a chat UI at /chat.
Can I run a model that is not in the default list?
Yes. You can add a model repository to run custom models with OpenLLM.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.