RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models models via Romanization

arXiv:2401.14280v2 Announce Type: replace
Abstract: This study addresses the challenge of extending Large Language Models (LLMs) to non-English languages using non-Roman scripts. We propose an approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance cross-lingual alignment. Our approach involves the continual pretraining of an English LLM like Llama 2 on romanized text of non-English, non-Roman script languages, followed by instruction tuning on romanized data. …

abstract arxiv capabilities challenge cs.ai cs.cl english form language language models languages large language large language models llms multilingual scripts study text tokens type via