Can Feedback Enhance Semantic Grounding in Large Vision-Language Models?

arXiv:2404.06510v1 Announce Type: new
Abstract: Enhancing semantic grounding abilities in Vision-Language Models (VLMs) often involves collecting domain-specific training data, refining the network architectures, or modifying the training recipes. In this work, we venture into an orthogonal direction and explore whether VLMs can improve their semantic grounding by "receiving" feedback, without requiring in-domain data, fine-tuning, or modifications to the network architectures. We systematically analyze this hypothesis using a feedback mechanism composed of a binary signal. We find that if prompted appropriately, …

arxiv cs.cv feedback language language models semantic type vision vision-language models