Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge

arXiv:2404.13292v1 Announce Type: new
Abstract: The popular subword tokenizers of current language models, such as Byte-Pair Encoding (BPE), are known not to respect morpheme boundaries, which affects the downstream performance of the models. While many improved tokenization algorithms have been proposed, their evaluation and cross-comparison is still an open problem. As a solution, we propose a combined intrinsic-extrinsic evaluation framework for subword tokenization. Intrinsic evaluation is based on our new UniMorph Labeller tool that classifies subword tokenization as either morphological …

abstract algorithms alien arxiv challenge comparison cs.ai cs.cl current encoding evaluation language language models performance popular tokenization type