Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models

arXiv:2403.15226v1 Announce Type: cross
Abstract: In this paper, we propose a novel parameter and computation efficient tuning method for Multi-modal Large Language Models (MLLMs), termed Efficient Attention Skipping (EAS). Concretely, we first reveal that multi-head attentions (MHAs), the main computational overhead of MLLMs, are often redundant to downstream tasks. Based on this observation, EAS evaluates the attention redundancy and skips the less important MHAs to speed up inference. Besides, we also propose a novel propagation-of-information adapter (PIA) to serve the …

abstract arxiv attention computation computational cs.cl cs.mm head language language models large language large language models mllms modal multi-head multi-modal novel paper transfer transfer learning type