Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models

arXiv:2403.00417v1 Announce Type: new
Abstract: Tokenization significantly influences language models(LMs)' performance. This paper traces the evolution of tokenizers from word-level to subword-level, analyzing how they balance tokens and types to enhance model adaptability while controlling complexity. Despite subword tokenizers like Byte Pair Encoding (BPE) overcoming many word tokenizer limitations, they encounter difficulties in handling non-Latin languages and depend heavily on extensive training data and computational resources to grasp the nuances of multiword expressions (MWEs). This article argues that tokenizers, more …

abstract adaptability arxiv balance complexity cs.cl encoding evolution language language models large language large language models limitations lms paper performance tokenization tokens traces type types word