CTranslate2 provides a C++ and Python inference engine for transformer models.
Open source
21 alternativesLLMKube
LLMKube runs self hosted LLM inference across heterogeneous GPU fleets using Kubernetes and several model runtimes.
About LLMKube
LLMKube is an open-source Kubernetes operator for self-hosted large language model inference. It is designed to run models across a heterogeneous GPU fleet, including NVIDIA CUDA, AMD Vulkan, Apple Silicon Metal, and Intel oneAPI / SYCL hardware. The project is licensed under Apache-2.0 and is intended for homelab and on-premises use.
You define a Model and an InferenceService in YAML, and the operator handles downloading, caching, GPU scheduling, health checks, scaling, and exposing an OpenAI-compatible API. It works with llama.cpp, vLLM, TGI, and mlx-server runtimes, and supports multi-GPU sharding. A ModelRouter component can route requests between local models and external providers such as Anthropic, OpenAI, and LiteLLM.
It suits teams and individuals who need to run LLMs on their own infrastructure for privacy, cost control, or air-gapped compliance. Use requires Kubernetes, with an optional metal-agent process for Apple Silicon hosts.
Key features
- Supports NVIDIA CUDA, AMD Vulkan, Apple Silicon Metal GPUs
- Works with llama.cpp, vLLM, TGI, mlx-server runtimes
- Provides OpenAI-compatible API endpoints
- Handles model downloading, caching and GPU scheduling
- Includes ModelRouter for local and external provider routing
- Optional Foreman add-on runs coding agents on local fleet
Best for: Teams and homelab users running LLMs on their own mixed-GPU Kubernetes clusters.
Good for
- Open source under Apache-2.0 license
- Supports heterogeneous GPU hardware in one setup
- Works with multiple common inference runtimes
- OpenAI-compatible endpoints for existing tools
Watch out for
- Requires Kubernetes to operate
- Apple Silicon hosts need separate metal-agent component
LLMKube at a glance
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 5, 2026 · Online when checked
- Listed
- Oct 5, 2026
LLMKube alternatives
Tools and sites you could use instead, ranked by how closely they match.
NobodyWho runs language models locally through an inference engine designed for efficient operation across devices.
Open source
22 alternatives
OpenLLM runs open-source language models as cloud API endpoints compatible with OpenAI clients.
Open source
21 alternatives
PowerInfer supports serving large language models for local deployment and inference.
Open source
23 alternatives
GPUStack manages GPU clusters for serving AI models and provides on demand GPU instances accessible through SSH.
Open source
21 alternatives
Rapid-MLX is an open-source LLM inference server for Apple Silicon, built on MLX, compatible with OpenAI and Anthropic APIs, focused on reliable tool calling for coding agents.
Open source
14 alternativesPopular in Model Serving & Inference
Well-known listings from the same topic.
MAI Playgroundplayground.microsoft.ai/chatBigModelbigmodel.cnkimi-k2.6platform.kimi.ai/docs/pricing/chat-k26Jinagithub.com/jina-ai/jinaFluxgithub.com/black-forest-labs/fluxprivateGPTgithub.com/imartinez/privateGPT
LLMKube FAQ
What is LLMKube?
LLMKube is a Kubernetes operator for self-hosted LLM inference across heterogeneous GPU fleets. You define Model and InferenceService resources and it manages deployment and API exposure.
Which GPUs and runtimes does LLMKube support?
It lists support for NVIDIA CUDA, AMD Vulkan, Apple Silicon Metal, and Intel oneAPI / SYCL. Supported runtimes include llama.cpp, vLLM, TGI, and mlx-server.
Does LLMKube work with external providers like OpenAI or Anthropic?
Yes, its ModelRouter can route between local models and external providers including Anthropic, OpenAI, and LiteLLM. It includes policy-aware routing with fail-closed semantics for regulated data.
Is LLMKube open source?
Yes, it is distributed under the Apache-2.0 license. The source code and documentation are hosted on GitHub.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.