PowerInfer supports serving large language models for local deployment and inference.
Open source
23 alternativesbitsandbytes
github.com/bitsandbytes-foundation/bitsandbytes
Model Serving & Inference
Open-source projectOpen source
bitsandbytes enables large language models in PyTorch using k-bit quantization.
About bitsandbytes
bitsandbytes is an open-source library that enables large language models in PyTorch through k-bit quantization. It offers three main features to reduce memory consumption for inference and training: 8-bit optimizers with block-wise quantization, LLM.int8() for 8-bit inference, and QLoRA for 4-bit training. It also provides quantization primitives such as Linear8bitLt and Linear4bit, plus an 8-bit optim module.
Key features
- 8-bit optimizers use block-wise quantization to reduce memory while maintaining 32-bit performance.
- LLM.int8() enables 8-bit LLM inference with half the required memory, without performance degradation.
- QLoRA quantizes models to 4-bit and adds trainable low-rank adaptation (LoRA) weights for training.
- Provides quantization primitives: Linear8bitLt, Linear4bit, and an 8-bit optim module.
- Supports CPU, NVIDIA, AMD, Intel GPUs and Intel Gaudi accelerators across Linux, Windows, macOS.
- Requires Python 3.10+ and PyTorch 2.4+ for all platforms.
Best for: Developers and researchers using PyTorch who need to reduce memory for large language model inference or training.
Good for
- Open source under the MIT license.
- Reduces memory consumption for inference and training through quantization.
- Supports a range of accelerators including CPU, NVIDIA GPU, AMD GPU, Intel GPU, and Intel Gaudi.
Watch out for
- Requires Python 3.10+ and PyTorch 2.4+.
- Some supported features may lack performance optimizations.
- On Intel Gaudi, QLoRA is partially supported and 8-bit optimizers are not supported; on macOS Metal, 8-bit optimizers are planned.
bitsandbytes at a glance
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
bitsandbytes alternatives
Tools and sites you could use instead, ranked by how closely they match.
ANEE wraps pretrained transformer inference with an energy budget and early exits inside model execution.
Open source
21 alternatives
Sonar provides large-scale inference for language models through the project hosted in this repository.
Open source
23 alternatives
OpenLLM runs open-source language models as cloud API endpoints compatible with OpenAI clients.
Open source
21 alternatives
NVIDIA Triton Inference Server provides an inference solution for deploying models in cloud and edge environments.
Open source
23 alternatives
Deploy generative and predictive AI models on Kubernetes with KServe’s distributed inference platform.
Open source
21 alternativesPopular in Model Serving & Inference
Well-known listings from the same topic.
MAI Playgroundplayground.microsoft.ai/chatBigModelbigmodel.cnkimi-k2.6platform.kimi.ai/docs/pricing/chat-k26Jinagithub.com/jina-ai/jinaFluxgithub.com/black-forest-labs/fluxprivateGPTgithub.com/imartinez/privateGPT
bitsandbytes FAQ
What are the minimum software requirements?
Python 3.10+ and PyTorch 2.4+.
Which operating systems are supported?
Linux with glibc >= 2.24, Windows 11/Server 2022+, and macOS 14+. Support varies by accelerator.
Is bitsandbytes open source?
Yes, it is MIT licensed.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.