Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models

arXiv:2402.14714v1 Announce Type: new
Abstract: This report introduces \texttt{EEVE-Korean-v1.0}, a Korean adaptation of large language models that exhibit remarkable capabilities across English and Korean text understanding. Building on recent highly capable but English-centric LLMs, such as SOLAR-10.7B and Phi-2, where non-English texts are inefficiently processed with English-centric tokenizers, we present an efficient and effective vocabulary expansion (EEVE) method, which encompasses parameter freezing and subword initialization. In contrast to previous efforts that believe new embeddings require trillions of training tokens, we …

abstract arxiv building capabilities cs.ai cs.cl english expansion language language models large language large language models llms multilingual phi phi-2 report solar text text understanding type understanding