Transformer Vs. MLP-Mixer Exponential Expressive Gap For NLP Problems. (arXiv:2208.08191v1 [cs.CL])

Vision-Transformers are widely used in various vision tasks. Meanwhile, there
is another line of works starting with the MLP-mixer trying to achieve similar
performance using mlp-based architectures. Interestingly, until now none
reported using them for NLP tasks, additionally until now non of those
mlp-based architectures claimed to achieve state-of-the-art in vision tasks. In
this paper, we analyze the expressive power of mlp-based architectures in
modeling dependencies between multiple different inputs simultaneously, and
show an exponential gap between the attention and …

arxiv gap mlp nlp transformer