Bridge the Modality and Capacity Gaps in Vision-Language Model Selection

arXiv:2403.13797v1 Announce Type: new
Abstract: Vision Language Models (VLMs) excel in zero-shot image classification by pairing images with textual category names. The expanding variety of Pre-Trained VLMs enhances the likelihood of identifying a suitable VLM for specific tasks. Thus, a promising zero-shot image classification strategy is selecting the most appropriate Pre-Trained VLM from the VLM Zoo, relying solely on the text data of the target dataset without access to the dataset's images. In this paper, we analyze two inherent challenges …

abstract arxiv bridge capacity classification cs.cv cs.lg excel image images language language model language models likelihood model selection specific tasks strategy tasks textual type vision vlm vlms zero-shot