The Cybertron Go package provides an interface for NLP technologies and pretrained Transformer models.
Open source
16 alternativesSentencePiece
SentencePiece provides unsupervised text tokenization for neural network based text generation.
About SentencePiece
SentencePiece is a text tokenizer and detokenizer for neural network-based text generation systems where vocabulary size is fixed before training. It implements Byte-Pair-Encoding and unigram language models and trains directly from raw sentences treated as Unicode characters.
It provides a Python module to encode text into subword pieces or vocabulary IDs and decode them back to the original string, with whitespace preserved using the ▁ symbol. It does not require language-specific pre- or post-processing and is not an official Google product.
Key features
- Train tokenizer directly from raw sentence files
- Encode text into subword pieces or vocabulary IDs
- Decode pieces or IDs back to original text
- Support Byte-Pair-Encoding and unigram language models
- Preserve whitespace with ▁ symbol for detokenization
- Sample different segmentations on the fly during training
Best for: Developers building neural text generation systems who need to train tokenizers from raw sentences.
Good for
- Lossless, reversible detokenization that preserves whitespace
- Training from raw sentences without language-specific pre-tokenizers
- Reported segmentation of around 50,000 sentences per second with ~6MB memory use
SentencePiece at a glance
- Category
- Development › Programming Languages
- Type
- Open-source project
- Pricing
- Not confirmed. Check the site
- Open source
- Yes · Source code
- Language
- English
- Last checked
- Oct 6, 2026 · Online when checked
- Listed
- Oct 5, 2026
SentencePiece alternatives
Tools and sites you could use instead, ranked by how closely they match.
This site compares programming languages and compilers through benchmark performance results.
Pick
16 alternatives
pgmpy provides Python tools for causal inference and probabilistic reasoning.
Open source
24 alternatives
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library.
Open source
24 alternatives
Python bindings for Frog expose Dutch NLP functions such as tagging, lemmatization, named entity recognition, and parsing.
Open source
24 alternatives
libpython-clj provides Python bindings that connect Python libraries with Clojure code.
Open source
24 alternativesPopular in Programming Languages
Well-known listings from the same topic.
JavaDecompilerjavadecompilers.comOpenNLPopennlp.apache.orgStandardJSstandardjs.comactnektosact.comRustPythonrustpython.github.ioOctaveoctave.org
SentencePiece FAQ
How do I install the Python module?
Install it via pip with: pip install sentencepiece
How do I train a model from a raw text file?
Use spm.SentencePieceTrainer.train with an input file, model_prefix, and vocab_size, without pre-tokenization.
How is whitespace handled for detokenization?
Whitespace is escaped as ▁ (U+2581) and included in tokens, so detokenization joins pieces and replaces ▁ with a space.
Is something wrong or out of date?
Tell us and an editor will check it.
Own this website?
Claim this listing
Claim the listing and suggest updates yourself. An editor checks every claim first.