Large Vocabulary Size Improves Large Language Models

arXiv:2406.16508v1 Announce Type: new
Abstract: This paper empirically investigates the relationship between subword vocabulary size and the performance of large language models (LLMs) to provide insights on how to define the vocabulary size. Experimental results show that larger vocabulary sizes lead to better performance in LLMs. Moreover, we consider a continual training scenario where a pre-trained language model is trained on a different target language. We introduce a simple method to use a new vocabulary instead of the pre-defined one. …

abstract arxiv continual cs.cl experimental insights language language models large language large language models llms paper performance relationship results show training type