What does self-attention learn from Masked Language Modelling?

Transformers are neural networks which revolutionised natural language processing and machine learning. They process sequences of inputs, like words, using a mechanism called self-attention, which is trained via masked language modelling (MLM). In MLM, a word is randomly masked in an input sequence, and the network is trained to predict the missing word. Despite the practical success of transformers, it remains unclear what type of data distribution self-attention can learn efficiently. Here, we show analytically that if one decouples the …

attention cond-mat.dis-nn cond-mat.stat-mech cs.cl inputs language language modelling language processing learn machine machine learning modelling natural natural language natural language processing network networks neural networks process processing self-attention stat.ml transformers via word words