Text Processing
Methods and techniques for analyzing, transforming, and managing text data for applications such as search, NLP, and automation.
posts
Tiny BPE Trainer – A Fast and Lightweight BPE Trainer in C++
Tiny BPE Trainer is a fast BPE trainer in C++17. Train your own tokenizer with this lightweight BPE trainer. HuggingFace-compatible output.
Continue Readingposts
A Fast, UTF-8 Aware C++ Tokenizer for NLP & ML
A blazing-fast, header-only C++ tokenizer with UTF-8 support, vocabulary encoding, and transformer-ready sequences, ideal for NLP and machine learning.
Continue Reading