GPT-NeoX implements model parallel autoregressive transformers on GPUs using Megatron and DeepSpeed libraries.
Open source
24 alternativesDeepSpeed-MII
MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
About DeepSpeed-MII
DeepSpeed-MII is an open-source Python library for model inference, powered by DeepSpeed-Inference. It allows non-persistent and persistent deployments in a few lines of code and automatically applies optimizations based on model architecture, model size, batch size, and available hardware.
It includes blocked KV caching, continuous batching, Dynamic SplitFuse, and tensor parallelism for text generation with models such as Llama-2, Mixtral, Phi-2, Qwen, Falcon, and OPT. Current releases use Hugging Face for model weights and tokenizers and are limited to listed families; legacy APIs cover additional models including BERT, RoBERTa, Stable Diffusion, Bloom, and GPT-J.
Key features
- Blocked KV caching for text generation inference
- Continuous batching for text generation workloads
- Dynamic SplitFuse for generation scheduling
- Tensor parallelism for multi-GPU inference
- Non-persistent and persistent deployments in code
- Text-to-image inference support for Stable Diffusion
Best for: Developers and teams deploying supported open-source language models for inference with Python.
Good for
- Open-source Python library for inference deployments
- Automatically selects optimizations based on model and hardware resources
- Documents support for over 37,000 models across listed architectures
Watch out for
- Current release limited to listed model families
- Uses Hugging Face backend for model weights and tokenizers
DeepSpeed-MII at a glance
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 5, 2026 · Online when checked
- Listed
- Oct 5, 2026
DeepSpeed-MII alternatives
Tools and sites you could use instead, ranked by how closely they match.
Extensible Toolkit for Finetuning and Inference of Large Foundation Models.
Open source
24 alternatives
The colibri-core NLP tool and library works with n-grams, skipgrams, and other linguistic patterns.
Open source
24 alternatives
Gymnasium provides a standard API for single agent reinforcement learning environments and related utilities.
Open source
24 alternatives
DEAP provides a Python framework for prototyping evolutionary computation algorithms and testing new ideas.
Open source
24 alternatives
DyNet provides tools for building dynamic neural networks and deep learning models.
Open source
24 alternativesPopular in AI Infrastructure & MLOps
Well-known listings from the same topic.
Guild AIguild.aiFlow AIflow-ai.comsystemprompt.iosystemprompt.ioKili Technologykili-technology.comAPI docsdocs.x.aiEyePop.aieyepop.ai
DeepSpeed-MII FAQ
How does MII work?
It is powered by DeepSpeed-Inference and automatically applies optimizations based on model architecture, model size, batch size, and available hardware resources.
Which models are supported?
Listed families include Falcon, Llama, Llama-2, Llama-3, Mistral, Mixtral, OPT, Phi-2, Qwen, and Qwen2.
Does MII support models other than LLMs?
Yes, it supports Stable Diffusion for text-to-image, and legacy APIs support BERT, RoBERTa, Bloom, GPT-J, and other models.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.