Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance

arXiv:2403.06265v1 Announce Type: cross
Abstract: Despite it being the cornerstone of BPE, the most common tokenization algorithm, the importance of compression in the tokenization process is still unclear. In this paper, we argue for the theoretical importance of compression, that can be viewed as 0-gram language modeling where equal probability is assigned to all tokens. We also demonstrate the empirical importance of compression for downstream success of pre-trained language models. We control the compression ability of several BPE tokenizers by …

abstract algorithm arxiv compression correlation cs.ai cs.cl cs.lg importance language modeling paper performance process text tokenization type