DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents

arXiv:2306.06306v3 Announce Type: replace
Abstract: Vision-language pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text. While existing vision-language pretraining models primarily focus on understanding single image associated with a single piece of text, they often ignore the alignment at the intra-document level, consisting of multiple sentences with multiple images. In this work, we propose DocumentCLIP, a salience-aware contrastive learning framework to enforce vision-language pretraining models to comprehend the interaction between images …

arxiv cs.ai cs.cv documents text type