VeCLIP: Improving CLIP Training via Visual-enriched Captions

arXiv:2310.07699v2 Announce Type: replace-cross
Abstract: Large-scale web-crawled datasets are fundamental for the success of pre-training vision-language models, such as CLIP. However, the inherent noise and potential irrelevance of web-crawled AltTexts pose challenges in achieving precise image-text alignment. Existing methods utilizing large language models (LLMs) for caption rewriting have shown promise on small, curated datasets like CC3M and CC12M. This study introduces a scalable pipeline for noisy caption rewriting. Unlike recent LLM rewriting techniques, we emphasize the incorporation of visual concepts …

arxiv captions clip cs.ai cs.cv cs.lg training type via visual