LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models

arXiv:2403.15388v1 Announce Type: new
Abstract: Large Multimodal Models (LMMs) have shown significant reasoning capabilities by connecting a visual encoder and a large language model. LMMs typically use a fixed amount of visual tokens, such as the penultimate layer features in the CLIP visual encoder, as the prefix content. Recent LMMs incorporate more complex visual inputs, such as high-resolution images and videos, which increase the number of visual tokens significantly. However, due to the design of the Transformer architecture, computational costs …

arxiv cs.ai cs.cl cs.cv large multimodal models llava multimodal multimodal models token type