Tiny BPE Trainer – A Fast and Lightweight BPE Trainer in C++

Thu, 07 Aug 2025 20:00:00 +0100

Introducing Tiny BPE Trainer

Most modern NLP models today from GPT to RoBERTa, rely on subword tokenization using Byte Pair Encoding (BPE). But what if you want to train your own vocabulary in pure C++?

Meet Tiny BPE Trainer - a blazing-fast, header-only BPE trainer written in modern C++17/20, with zero dependencies, full UTF-8 support, and HuggingFace-compatible output (vocab.txt, merges.txt).

Why Another BPE Trainer?

Because existing options are often:

Python-only, with heavy runtime dependencies (Rust, Protobuf, etc.)
Not easily embeddable in C++ applications
Not focused on speed, simplicity, or cross-platform usage

Tiny BPE Trainer is:

A Fast, UTF-8 Aware C++ Tokenizer for NLP & ML

Wed, 06 Aug 2025 06:00:00 +0100

Introducing Modern Text Tokenizer

Modern natural language processing (NLP) models like BERT, DistilBERT, and other transformer-based architectures rely heavily on effective tokenization. But C++ developers often face limited options like bloated dependencies, poor Unicode support, or lack of compatibility with vocab-based encoders.

That’s why I created Modern Text Tokenizer - a blazing-fast, header-only C++ tokenizer that’s UTF-8 aware, zero-dependency, and ML-ready out of the box.

What Makes It Unique?

Zero Dependencies – No Boost, no ICU, no external libs.
UTF-8 Safe – Correctly handles multilingual text, emojis, and multibyte characters.
Header-Only – Drop it into your project and go.
Vocabulary Encoding – Load vocab.txt from HuggingFace and generate token IDs.
Transformer-Ready – Supports [CLS], [SEP], [PAD], and sequence formatting.

Key Features

Fast ASCII vs Unicode branching using std::string_view

Fluent API for configuration:

1TextTokenizer tokenizer;
2tokenizer
3.set_lowercase(true)
4.set_split_on_punctuation(true)
5.set_keep_punctuation(true);

Load vocabularies:
```
1tokenizer.load_vocab("vocab.txt");
```

Encode / Decode:

1auto ids = tokenizer.encode("Hello world!");
2std::string decoded = tokenizer.decode(ids);

Performance

1Performance test with 174000 characters
2
3Results:
4Tokenization: 2159 μs (22000 tokens)
5Encoding: 1900 μs
6Decoding: 430 μs
7Total time: 4.49 ms
8Throughput: 36.97 MB/s

Benchmarked on Ryzen 9 5900X @ -O3 in release mode.

Tokenizer on [ MECANIK DEV ]