Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding

arXiv:2312.00081v2 Announce Type: replace
Abstract: Vision language models (VLM) have demonstrated remarkable performance across various downstream tasks. However, understanding fine-grained visual-linguistic concepts, such as attributes and inter-object relationships, remains a significant challenge. While several benchmarks aim to evaluate VLMs in finer granularity, their primary focus remains on the linguistic aspect, neglecting the visual dimension. Here, we highlight the importance of evaluating VLMs from both a textual and visual perspective. We introduce a progressive pipeline to synthesize images that vary in …

arxiv cs.cv fine-grained language language understanding type understanding vision