Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie Embedding

arXiv:2404.00862v1 Announce Type: new
Abstract: Large language models (LLMs) have demonstrated exceptional performance in various NLP applications. However, the majority of existing open-source LLMs are pre-trained primarily on English data and little part of other languages. This deficiency in multilingual training data results in suboptimal performance when applied to languages with fewer available resources. Furthermore, enhancing the performance of LLMs on low-resource languages by full-parameter fine-tuning with additional data requires substantial computational resources, posing computational barriers for research organizations and …

abstract applications arxiv bilingual cs.ai cs.cl data embedding english however language language models languages large language large language models llms multilingual nlp part performance qlora results training training data transfer transfer learning type zip