AceGPT, Localizing Large Language Models in Arabic

arXiv:2309.12053v5 Announce Type: replace
Abstract: This paper is devoted to the development of a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models. Significant concerns emerge when addressing cultural sensitivity and local values. To address this, the paper proposes a comprehensive solution that includes further pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic, alongside Reinforcement Learning with AI Feedback (RLAIF) …

abstract arabic arxiv concerns cs.cl current development language language model language models large language large language model large language models llm paper sensitivity type values