What matters when building vision-language models?

arXiv:2405.02246v1 Announce Type: new
Abstract: The growing interest in vision-language models (VLMs) has been driven by improvements in large language models and vision transformers. Despite the abundance of literature on this subject, we observe that critical decisions regarding the design of VLMs are often not justified. We argue that these unsupported decisions impede progress in the field by making it difficult to identify which choices improve model performance. To address this issue, we conduct extensive experiments around pre-trained models, architecture …

abstract arxiv building cs.ai cs.cv decisions design improvements language language models large language large language models literature observe progress transformers type vision vision-language vision-language models vision transformers vlms