Exploiting Pseudo Image Captions for Multimodal Summarization

arXiv:2305.05496v2 Announce Type: replace
Abstract: Cross-modal contrastive learning in vision language pretraining (VLP) faces the challenge of (partial) false negatives. In this paper, we study this problem from the perspective of Mutual Information (MI) optimization. It is common sense that InfoNCE loss used in contrastive learning will maximize the lower bound of MI between anchors and their positives, while we theoretically prove that MI involving negatives also matters when noises commonly exist. Guided by a more general lower bound form …

abstract arxiv captions challenge common sense cs.cl false image information language loss modal multimodal optimization paper perspective pretraining sense study summarization type vision will