ML.NET provides an open source, cross platform machine learning framework for .NET applications.
Open source
24 alternativesshimmytok
The shimmytok project provides a pure Rust GGUF tokenizer compatible with llama.cpp and ready for WebAssembly.
About shimmytok
shimmytok is a Rust library that reads the tokenizer embedded in GGUF model files and encodes text to tokens and decodes tokens back to text, without Python, C++ bindings, or separate tokenizer files.
It handles SentencePiece, BPE, WordPiece, Unigram, and RWKV tokenizers, offers batch encoding with optional Rayon parallelism and token-by-token decoding, and supports WASM/WASI builds with a sequential backend. The project reports token-for-token matching output with llama.cpp for the models listed in its validation table.
Key features
- Loads tokenizer directly from .gguf model files
- Encodes text to tokens and decodes tokens to text
- Supports SentencePiece, BPE, WordPiece, Unigram and RWKV
- Provides batch encoding with optional Rayon parallelism
- Provides token-by-token decoding for streaming output
- Builds for WASM/WASI with sequential backend
Best for: Rust developers working with GGUF models who need tokenization without C++ bindings or separate tokenizer files.
Good for
- No C++ dependencies or separate tokenizer files required
- Tested token-for-token against llama.cpp for listed models
- Zero unsafe code in critical paths according to project
shimmytok at a glance
- Category
- Development › Programming Languages
- Type
- Open-source project
- Pricing
- Free
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
shimmytok alternatives
Tools and sites you could use instead, ranked by how closely they match.
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library.
Open source
24 alternatives
rb-libsvm provides Ruby bindings for the LIBSVM machine learning library.
Open source
24 alternativesLMQL provides a programming language for interactions with large language models.
The MLJ project provides machine learning tools and a framework for Julia.
Treat supplies Ruby developers with a framework for natural language processing.
Popular in Programming Languages
Well-known listings from the same topic.
Language Benchmarksprogramming-language-benchmarks.vercel.appJavaDecompilerjavadecompilers.comOpenNLPopennlp.apache.orgStandardJSstandardjs.comactnektosact.comRustPythonrustpython.github.io
shimmytok FAQ
How do I use shimmytok with a model file?
Use Tokenizer::from_gguf_file with the path to a .gguf file, then call encode and decode methods.
Which tokenizer types does it handle?
SentencePiece, BPE, WordPiece, Unigram, and RWKV, including 41 BPE pre-tokenization patterns.
Does it need Python, C++, or separate tokenizer files?
No. It extracts and runs the tokenizer embedded directly in the GGUF file.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.