WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. (arXiv:2112.06598v2 [cs.CL] UPDATED)

Web: http://arxiv.org/abs/2112.06598

Large pretrained language models (LMs) have become the central building block
of many NLP applications. Training these models requires ever more
computational resources and most of the existing models are trained on English
text only. It is exceedingly expensive to train these models in other
languages. To alleviate this problem, we introduce a novel method -- called
WECHSEL -- to efficiently and effectively transfer pretrained LMs to new
languages. WECHSEL can be applied to any model which uses subword-based
tokenization …

arxiv cross cross-lingual language language models models transfer