Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

arXiv:2404.17790v1 Announce Type: new
Abstract: Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an LLM with enhanced Japanese capability, by extending the vocabulary of Llama 2 to include Japanese characters and conducting continual pre-training on a large Japanese web corpus. Experimental results confirmed that the performance on Japanese tasks drastically improved through continual …

abstract arxiv capabilities capability continual cost cross-lingual cs.ai cs.cl english english language japanese language language models language resources large language large language models llm llms pre-training reduce resources study training type vast