Iterated Learning Improves Compositionality in Large Vision-Language Models

arXiv:2404.02145v2 Announce Type: replace
Abstract: A fundamental characteristic common to both human vision and natural language is their compositional nature. Yet, despite the performance gains contributed by large vision and language pretraining, recent investigations find that most-if not all-our state-of-the-art vision-language models struggle at compositionality. They are unable to distinguish between images of " a girl in white facing a man in black" and "a girl in black facing a man in white". Moreover, prior work suggests that compositionality doesn't …

abstract art arxiv contributed cs.cv human images investigations language language models natural natural language nature performance pretraining state struggle type vision vision-language vision-language models