Theoretical Foundations of Deep Selective State-Space Models

arXiv:2402.19047v1 Announce Type: new
Abstract: Structured state-space models (SSMs) such as S4, stemming from the seminal work of Gu et al., are gaining popularity as effective approaches for modeling sequential data. Deep SSMs demonstrate outstanding performance across a diverse set of domains, at a reduced training and inference cost compared to attention-based transformers. Recent developments show that if the linear recurrence powering SSMs allows for multiplicative interactions between inputs and hidden states (e.g. GateLoop, Mamba, GLA), then the resulting architecture …

abstract arxiv attention cost cs.lg data diverse domains inference math.ds modeling performance set space state stemming training transformers type work