Sonar provides large-scale inference for language models through the project hosted in this repository.
Open source
23 alternativesbitnet.cpp
BitNet provides an inference framework for one-bit large language models.
About bitnet.cpp
bitnet.cpp is an inference framework for 1-bit large language models such as BitNet b1.58. It provides kernels for inference on CPU and GPU, with NPU support listed as coming next. It can be tried through an online demo or built and run locally.
The repository includes the BitNet-b1.58-2B-4T language model and BitNet embedding models, along with scripts for running inference and an inference server. Documentation reports running a 100B BitNet b1.58 model on a single CPU at 5-7 tokens per second.
Key features
- Runs 1.58-bit LLMs such as BitNet b1.58 on CPU and GPU
- Runs 100B-parameter BitNet b1.58 model on single CPU
- Provides conversational mode for interactive use
- Offers GPU inference kernel for 1-bit models
- Supports I2_S conversion for BitNet embedding models on x86 CPUs
- Includes scripts to run inference and run inference server
Best for: Developers and researchers who want to run 1-bit language or embedding models on local CPUs or GPUs.
Good for
- Reported speedups of 2.37x to 6.17x on x86 and 1.37x to 5.07x on ARM CPUs
- Reported energy reductions of 71.9% to 82.2% on x86 and 55.4% to 70.0% on ARM
- Open source repository with local CPU and GPU build and run scripts
Watch out for
- NPU support is listed as coming next and is not currently available
bitnet.cpp at a glance
- Website
- github.com/microsoft/BitNet
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
bitnet.cpp alternatives
Tools and sites you could use instead, ranked by how closely they match.
PowerInfer supports serving large language models for local deployment and inference.
Open source
23 alternatives
bitsandbytes enables large language models in PyTorch using k-bit quantization.
Open source
21 alternatives
MindSpore provides a deep learning framework for training and inference across mobile, edge, and cloud scenarios.
Open source
21 alternatives
NobodyWho runs language models locally through an inference engine designed for efficient operation across devices.
Open source
22 alternatives
OpenLLM runs open-source language models as cloud API endpoints compatible with OpenAI clients.
Open source
21 alternativesPopular in Model Serving & Inference
Well-known listings from the same topic.
MAI Playgroundplayground.microsoft.ai/chatBigModelbigmodel.cnkimi-k2.6platform.kimi.ai/docs/pricing/chat-k26Jinagithub.com/jina-ai/jinaFluxgithub.com/black-forest-labs/fluxprivateGPTgithub.com/imartinez/privateGPT
bitnet.cpp FAQ
What hardware does bitnet.cpp support?
It supports inference on CPU and GPU, with NPU support listed as coming next. It can be used via an online demo or built and run on a local CPU or GPU.
Which models are included?
It includes BitNet-b1.58-2B-4T, described as a 2.4B-parameter model trained on 4 trillion tokens, and BitNet-embedding-0.6B and 270M embedding models.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.