BRAVE: Broadening the visual encoding of vision-language models

arXiv:2404.07204v1 Announce Type: cross
Abstract: Vision-language models (VLMs) are typically composed of a vision encoder, e.g. CLIP, and a language model (LM) that interprets the encoded features to solve downstream tasks. Despite remarkable progress, VLMs are subject to several shortcomings due to the limited capabilities of vision encoders, e.g. "blindness" to certain image features, visual hallucination, etc. To address these issues, we study broadening the visual encoding capabilities of VLMs. We first comprehensively benchmark several vision encoders with different inductive …

abstract arxiv blindness brave capabilities clip cs.ai cs.cv cs.lg encoder encoding features language language model language models progress solve tasks type vision vision-language models visual vlms