Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Features

Unveiling the reasons behind the exceptional success of transformers requires a better understanding of why attention layers are suitable for NLP tasks. In particular, such tasks require predictive models to capture contextual meaning which often depends on one or few words, even if the sentence is long. Our work studies this key property, dubbed word sensitivity (WS), in the prototypical setting of random features. We show that attention layers enjoy high WS, namely, there exists a vector in the space …

attention cs.lg features meaning nlp predictive predictive models random sensitivity stat.ml study success tasks transformers understanding via word words