Understanding MLP-Mixer as a Wide and Sparse MLP

arXiv:2306.01470v2 Announce Type: replace
Abstract: Multi-layer perceptron (MLP) is a fundamental component of deep learning, and recent MLP-based architectures, especially the MLP-Mixer, have achieved significant empirical success. Nevertheless, our understanding of why and how the MLP-Mixer outperforms conventional MLPs remains largely unexplored. In this work, we reveal that sparseness is a key mechanism underlying the MLP-Mixers. First, the Mixers have an effective expression as a wider MLP with Kronecker-product weights, clarifying that the Mixers efficiently embody several sparseness properties explored …

abstract architectures arxiv cs.lg deep learning fundamental key layer mlp perceptron stat.ml success type understanding work