ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model

arXiv:2404.02534v1 Announce Type: new
Abstract: In recent years, the development of pre-trained language models (PLMs) has gained momentum, showcasing their capacity to transcend linguistic barriers and facilitate knowledge transfer across diverse languages. However, this progress has predominantly bypassed the inclusion of very-low resource languages, creating a notable void in the multilingual landscape. This paper addresses this gap by introducing four tailored PLMs specifically finetuned for Angolan languages, employing a Multilingual Adaptive Fine-tuning (MAFT) approach. In this paper, we survey the …

abstract arxiv capacity cs.ai cs.cl data development diverse embedding however inclusion knowledge language language model language models languages low progress synthetic synthetic data transfer type