Two Counterexamples to \textit{Tokenization and the Noiseless Channel}

arXiv:2402.14614v1 Announce Type: new
Abstract: In \textit{Tokenization and the Noiseless Channel} \cite{zouhar-etal-2023-tokenization}, R\'enyi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest R\'enyi efficiency of the unigram distribution should be chosen. The R\'enyi efficiency is thus treated as a predictor of downstream performance (e.g., predicting BLEU for a machine translation task), without the expensive step of training multiple models with different tokenizers. Although useful, the predictive power of …

abstract arxiv cs.cl distribution efficiency intrinsic leads nlp tasks tokenization type