[R] Is Mamba Capable of In-Context Learning?

*Link:* [https://arxiv.org/abs/2402.03170](https://arxiv.org/abs/2402.03170)

*Authors:* Riccardo Grazzi\*, Julien Siems\*, Simon Schrodi, Thomas Brox, Frank Hutter

\*equal contribution

*Abstract:* This work provides empirical evidence that Mamba, a newly proposed selective structured state space model, has similar in-context learning (ICL) capabilities as transformers. We evaluated Mamba on tasks involving simple function approximation as well as more complex natural language processing problems. Our results demonstrate that across both categories of tasks, Mamba matches the performance of transformer models for ICL. Further analysis reveals that like …

abstract approximation authors capabilities context evidence function in-context learning language language processing machinelearning mamba natural natural language natural language processing processing simple space state tasks thomas transformers work