Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection

arXiv:2403.07883v1 Announce Type: new
Abstract: Vision Transformers (ViTs) have become increasingly popular in large-scale Vision and Language Pre-training (VLP) models. Although previous VLP research has demonstrated the efficacy of ViTs, these efforts still struggle with computational inefficiencies caused by lengthy visual sequences. To address this challenge, we introduce an efficient VLP approach called TRIPS, which stands for Text-Relevant Image Patch Selection. TRIPS progressively reduces the visual sequence using a text-guided patch-selection layer in the visual backbone, thereby accelerating both training …

abstract arxiv become challenge computational cs.ai cs.cv image language popular pre-training research scale struggle text training transformers type vision vision-and-language vision transformers visual