DeciMamba: Exploring the Length Extrapolation Potential of Mamba

arXiv:2406.14528v1 Announce Type: new
Abstract: Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrates high performance and achieves Transformer-level capabilities while requiring substantially fewer computational resources. In this paper we explore the length-generalization capabilities of Mamba, which we find to be relatively limited. Through a series of visualizations and analyses we identify that the limitations arise from a restricted effective receptive field, dictated by the …

abstract alternative arxiv capabilities challenge complexity computational cs.ai cs.lg explore input mamba paper performance potential processing resources transformer transformers type while