A Generic Shared Attention Mechanism for Various Backbone Neural Networks

arXiv:2210.16101v2 Announce Type: replace
Abstract: The self-attention mechanism has emerged as a critical component for improving the performance of various backbone neural networks. However, current mainstream approaches individually incorporate newly designed self-attention modules (SAMs) into each layer of the network for granted without fully exploiting their parameters' potential. This leads to suboptimal performance and increased parameter consumption as the network depth increases. To improve this paradigm, in this paper, we first present a counterintuitive but inherent phenomenon: SAMs tend to …

abstract arxiv attention cs.ai cs.cv current however improving layer leads modules network networks neural networks parameters performance self-attention type