Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference

arXiv:2405.05803v1 Announce Type: new
Abstract: Multimodal large language models (MLLMs) demand considerable computations for inference due to the extensive parameters and the additional input tokens needed for visual information representation. Herein, we introduce Visual Tokens Withdrawal (VTW), a plug-and-play module to boost MLLMs for rapid inference. Our approach is inspired by two intriguing phenomena we have observed: (1) the attention sink phenomenon that is prevalent in LLMs also persists in MLLMs, suggesting that initial tokens and nearest tokens receive the …

arxiv boosting cs.ai cs.cv inference language language models large language large language models multimodal tokens type visual