MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts

arXiv:2401.04081v2 Announce Type: replace
Abstract: State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based Large Language Models, including recent state-of-the-art open models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable performance. Our model, MoE-Mamba, outperforms both Mamba …

abstract art arxiv become cs.ai cs.cl cs.lg experts language language models large language large language models mamba mixture of experts modeling moe open models space state transformer transformers type