If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions

arXiv:2403.16442v1 Announce Type: cross
Abstract: Recent works often assume that Vision-Language Model (VLM) representations are based on visual attributes like shape. However, it is unclear to what extent VLMs prioritize this information to represent concepts. We propose Extract and Explore (EX2), a novel approach to characterize important textual features for VLMs. EX2 uses reinforcement learning to align a large language model with VLM preferences and generates descriptions that incorporate the important features for the VLM. Then, we inspect the descriptions …

abstract arxiv clip concept concepts cs.cl cs.cv cs.lg explore extract however information language language model novel talk through type understanding vision visual vlm vlms