The Illusion of State in State-Space Models

arXiv:2404.08819v1 Announce Type: new
Abstract: State-space models (SSMs) have emerged as a potential alternative architecture for building large language models (LLMs) compared to the previously ubiquitous transformer architecture. One theoretical weakness of transformers is that they cannot express certain kinds of sequential computation and state tracking (Merrill and Sabharwal, 2023), which SSMs are explicitly designed to address via their close architectural similarity to recurrent neural networks (RNNs). But do SSMs truly have an advantage (over transformers) in expressive power for …

abstract architecture arxiv building computation cs.cc cs.cl cs.fl cs.lg express language language models large language large language models llms space ssms state tracking transformer transformer architecture transformers type