ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs

arXiv:2406.08164v1 Announce Type: new
Abstract: Compositional Reasoning (CR) entails grasping the significance of attributes, relations, and word order. Recent Vision-Language Models (VLMs), comprising a visual encoder and a Large Language Model (LLM) decoder, have demonstrated remarkable proficiency in such reasoning tasks. This prompts a crucial question: have VLMs effectively tackled the CR challenge? We conjecture that existing CR benchmarks may not adequately push the boundaries of modern VLMs due to the reliance on an LLM-only negative text generation pipeline. Consequently, …

abstract arxiv attributes cs.cv decoder encoder evaluation grasping language language model language models large language large language model llm modern prompts question reasoning relations significance tasks type vision vision-language vision-language models visual vlms word