KubeAI

KubeAI serves machine learning models on Kubernetes, including speech to text, language, vision, and embedding models.

github.com/substratusai/kubeai
KubeAI homepage screenshot

About KubeAI

KubeAI is an AI inference operator for Kubernetes that deploys and scales machine learning models, including LLMs, vision-language models, embeddings, reranking, and speech-to-text. It uses a model operator to manage backend server pods, model downloads, volume mounts, and LoRA adapters, plus a proxy that provides an OpenAI-compatible API with request queueing and retries.

It supports vLLM and Ollama for LLMs, FasterWhisper for transcription, and Infinity for embeddings. It runs on CPU, GPU, or TPU in a Kubernetes cluster and does not require Istio, Knative, or a Prometheus metrics adapter.

Key features

  • Operate vLLM and Ollama servers for LLM inference
  • Transcribe audio with FasterWhisper via audio transcription endpoint
  • Generate vector embeddings with Infinity models
  • Reorder search results with cross-encoder reranking models
  • Scale model replicas from zero based on demand
  • Provide OpenAI-compatible API for chat, embeddings, and transcription
Best for: Teams and operators that serve LLMs, embeddings, and speech-to-text models on Kubernetes.
Good for
  • Operates without requiring Istio, Knative, or Prometheus metrics adapter
  • Supports CPU, GPU, and TPU hardware
  • Queues requests during scale-from-zero and retries failed backends

KubeAI at a glance

Type
Open-source project
Pricing
Not confirmed. Check the site
Open source
Yes · Source code
Language
English
Last checked
Oct 5, 2026 · Online when checked
Listed
Oct 5, 2026

KubeAI alternatives

Tools and sites you could use instead, ranked by how closely they match.

All 24 alternatives

Popular in AI Speech Transcription

Well-known listings from the same topic.

All 43

KubeAI FAQ

What do I need to run KubeAI?
A Kubernetes cluster with CPU, GPU, or TPU resources. It does not require Istio, Knative, or a Prometheus metrics adapter.
Does KubeAI work with OpenAI client libraries?
Yes. It exposes OpenAI-compatible endpoints including chat completions, completions, embeddings, rerank, models, and audio transcriptions.
How does KubeAI handle scaling and routing?
It scales replicas from zero, queues requests while scaling, retries failed backends, and uses prefix-aware load balancing for backend serving engines.
Is something wrong or out of date?

Tell us and an editor will check it.

Own this website?

Claim the listing and suggest updates yourself. An editor checks every claim first.

Claim this listing