Can linguistically informed tokenization improve language modeling for morphologically rich, low-resource languages?
Compared BPE and morpheme-based tokenization using comparable GPT-2 configurations trained on a 100 MB Hindi corpus. The experiment evaluated how linguistic decomposition affects perplexity and vocabulary coverage under limited computational resources.
18.7
Morpheme-model perplexity
24.3
BPE-model perplexity
23%
Lower perplexity in the evaluated setup
