iMixer: hierarchical Hopfield network implies an invertible, implicit and iterative MLP-Mixer

arXiv:2304.13061v2 Announce Type: replace
Abstract: In the last few years, the success of Transformers in computer vision has stimulated the discovery of many alternative models that compete with Transformers, such as the MLP-Mixer. Despite their weak inductive bias, these models have achieved performance comparable to well-studied convolutional neural networks. Recent studies on modern Hopfield networks suggest the correspondence between certain energy-based associative memory models and Transformers or MLP-Mixer, and shed some light on the theoretical background of the Transformer-type architectures …

abstract arxiv bias computer computer vision cond-mat.dis-nn convolutional neural networks cs.cv cs.lg cs.ne discovery hierarchical inductive iterative mlp network networks neural networks performance success transformers type vision