Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models. (arXiv:2205.11758v1 [cs.CL])

The emergent cross-lingual transfer seen in multilingual pretrained models
has sparked significant interest in studying their behavior. However, because
these analyses have focused on fully trained multilingual models, little is
known about the dynamics of the multilingual pretraining process. We
investigate when these models acquire their in-language and cross-lingual
abilities by probing checkpoints taken from throughout XLM-R pretraining, using
a suite of linguistic tasks. Our analysis shows that the model achieves high
in-language performance early on, with lower-level linguistic skills …

arxiv cross-lingual dynamics language language models