Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models

arXiv:2402.16438v1 Announce Type: new
Abstract: Large language models (LLMs) demonstrate remarkable multilingual capabilities without being pre-trained on specially curated multilingual parallel corpora. It remains a challenging problem to explain the underlying mechanisms by which LLMs process multilingual texts. In this paper, we delve into the composition of Transformer architectures in LLMs to pinpoint language-specific regions. Specially, we propose a novel detection method, language activation probability entropy (LAPE), to identify language-specific neurons within LLMs. Based on LAPE, we conduct comprehensive experiments …

abstract architectures arxiv capabilities cs.cl key language language models large language large language models llms multilingual neurons paper process the key transformer type