[R] MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts

**Paper**: [https://arxiv.org/abs/2401.04081](https://arxiv.org/abs/2401.04081)

**Code**: [https://github.com/llm-random/llm-random](https://github.com/llm-random/llm-random)

**Abstract**:

>State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based LLMs, including recent state-of-the-art open-source models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable, Transformer-like performance. Our model, **MoE-Mamba**, outperforms both Mamba and …

abstract art become experts llms machinelearning mamba mixture of experts modeling moe open-source models scaling space state transformer transformers