Jamba: A Hybrid Transformer-Mamba Language Model

arXiv:2403.19887v1 Announce Type: cross
Abstract: We present Jamba, a new base large language model based on a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. Specifically, Jamba interleaves blocks of Transformer and Mamba layers, enjoying the benefits of both model families. MoE is added in some of these layers to increase model capacity while keeping active parameter usage manageable. This flexible architecture allows resource- and objective-specific configurations. In the particular configuration we have implemented, we end up with a powerful model that …

abstract architecture arxiv benefits capacity cs.cl cs.lg experts families hybrid jamba language language model large language large language model mamba moe novel transformer type