State Space Models as Foundation Models: A Control Theoretic Overview

arXiv:2403.16899v1 Announce Type: cross
Abstract: In recent years, there has been a growing interest in integrating linear state-space models (SSM) in deep neural network architectures of foundation models. This is exemplified by the recent success of Mamba, showing better performance than the state-of-the-art Transformer architectures in language tasks. Foundation models, like e.g. GPT-4, aim to encode sequential data into a latent space in order to learn a compressed representation of the data. The same goal has been pursued by control …

abstract architectures art arxiv control cs.cl cs.lg cs.sy deep neural network eess.sy foundation language linear mamba network neural network overview performance space state success tasks transformer type