FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models

arXiv:2405.10286v1 Announce Type: new
Abstract: Despite noise and caption quality having been acknowledged as important factors impacting vision-language contrastive pre-training, in this paper, we show that the full potential of improving the training process by addressing such issues is yet to be realized. Specifically, we firstly study and analyze two issues affecting training: incorrect assignment of negative pairs, and low caption quality and diversity. Then, we devise effective solutions for addressing both problems, which essentially require training with multiple true …

abstract arxiv cs.ai cs.cv improving language language models noise paper pre-training process quality results show study training type vision vision-language vision-language models