State Space Model for New-Generation Network Alternative to Transformers: A Survey

arXiv:2404.09516v1 Announce Type: new
Abstract: In the post-deep learning era, the Transformer architecture has demonstrated its powerful performance across pre-trained big models and various downstream tasks. However, the enormous computational demands of this architecture have deterred many researchers. To further reduce the complexity of attention models, numerous efforts have been made to design more efficient methods. Among them, the State Space Model (SSM), as a possible replacement for the self-attention based Transformer model, has drawn more and more attention in …

arxiv cs.ai cs.cl cs.cv cs.lg cs.mm network space state state space model survey transformers type