Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages

arXiv:2402.12204v1 Announce Type: new
Abstract: While large language models (LLMs) have been pre-trained on multilingual corpora, their performance still lags behind in most languages compared to a few resource-rich languages. One common approach to mitigate this issue is to translate training data from resource-rich languages into other languages and then continue training. However, using the data obtained solely relying on translation while ignoring the original capabilities of LLMs across languages is not always effective, which we show will limit the …

abstract arxiv capabilities cs.cl data distillation issue language language models languages large language large language models llms multilingual performance through training training data translate type