BUS:Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarization

arXiv:2307.08504v2 Announce Type: replace
Abstract: Vision Transformer (ViT) based Vision-Language Pre-training (VLP) models have demonstrated impressive performance in various tasks. However, the lengthy visual token sequences fed into ViT can lead to training inefficiency and ineffectiveness. Existing efforts address the challenge by either bottom-level patch extraction in the ViT backbone or top-level patch abstraction outside, not balancing training efficiency and effectiveness well. Inspired by text summarization in natural language processing, we propose a Bottom-Up Patch Summarization approach named BUS, coordinating …

abstract arxiv challenge cs.cv extraction fed language performance pre-training summarization tasks token training transformer type vision visual vit