Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge

arXiv:2403.05189v1 Announce Type: new
Abstract: Acquiring factual knowledge for language models (LMs) in low-resource languages poses a serious challenge, thus resorting to cross-lingual transfer in multilingual LMs (ML-LMs). In this study, we ask how ML-LMs acquire and represent factual knowledge. Using the multilingual factual knowledge probing dataset, mLAMA, we first conducted a neuron investigation of ML-LMs (specifically, multilingual BERT). We then traced the roots of facts back to the knowledge source (Wikipedia) to identify the ways in which ML-LMs acquire …

abstract arxiv challenge cross-lingual cs.ai cs.cl facts independent knowledge language language models languages lms low multilingual study tracing transfer type