Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer

arXiv:2404.04042v1 Announce Type: new
Abstract: This paper explores cost-efficient methods to adapt pretrained Large Language Models (LLMs) to new lower-resource languages, with a specific focus on Estonian. Leveraging the Llama 2 model, we investigate the impact of combining cross-lingual instruction-tuning with additional monolingual pretraining. Our results demonstrate that even a relatively small amount of additional monolingual pretraining followed by cross-lingual instruction-tuning significantly enhances results on Estonian. Furthermore, we showcase cross-lingual knowledge transfer from high-quality English instructions to Estonian, resulting in …

abstract adapt arxiv cost cross-lingual cs.cl focus impact knowledge language language models languages large language large language models llama llama 2 llama 2 model llms paper pretraining results teaching through transfer type