Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning

arXiv:2402.17510v1 Announce Type: new
Abstract: Vision-language models (VLMs) mainly rely on contrastive training to learn general-purpose representations of images and captions. We focus on the situation when one image is associated with several captions, each caption containing both information shared among all captions and unique information per caption about the scene depicted in the image. In such cases, it is unclear whether contrastive losses are sufficient for learning task-optimal representations that contain all the information provided by the captions or …

abstract arxiv captions cs.ai cs.cv focus general image images information language language models learn per representation representation learning training type vision vision-language models vlms