VL-Mamba: Exploring State Space Models for Multimodal Learning

arXiv:2403.13600v1 Announce Type: new
Abstract: Multimodal large language models (MLLMs) have attracted widespread interest and have rich applications. However, the inherent attention mechanism in its Transformer structure requires quadratic complexity and results in expensive computational overhead. Therefore, in this work, we propose VL-Mamba, a multimodal large language model based on state space models, which have been shown to have great potential for long-sequence modeling with fast inference and linear scaling in sequence length. Specifically, we first replace the transformer-based backbone …

abstract applications arxiv attention complexity computational cs.cv however language language model language models large language large language model large language models mamba mllms multimodal multimodal large language model multimodal learning results space state transformer type work