The challenge
Tokenisation is a foundation for training and using language models. Underrepresented scripts can suffer from inefficient segmentation.
Approach
- 01Prepare Balochi text samples for tokenizer training.
- 02Train and assess a Unigram tokenisation configuration.
- 03Measure unknown-token behaviour and character coverage.
Documented outcome
The repository reports 99.9% character coverage and a 0.0% unknown-token rate in its documented evaluation.
Interpretation & limitations
Coverage measures describe the assessed text, not a guarantee across all future sources and writing conventions.