CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers

arXiv:2305.17455v4 Announce Type: replace-cross
Abstract: Recent vision-language models have achieved tremendous advances. However, their computational costs are also escalating dramatically, making model acceleration exceedingly critical. To pursue more efficient vision-language Transformers, this paper introduces Cross-Guided Ensemble of Tokens (CrossGET), a general acceleration framework for vision-language Transformers. This framework adaptively combines tokens in real-time during inference, significantly reducing computational costs while maintaining high performance. CrossGET features two primary innovations: 1) Cross-Guided Matching and Ensemble. CrossGET leverages cross-modal guided token matching and …

arxiv cs.cl cs.cv ensemble language replace tokens transformers type vision vision-language