CapsFusion: Rethinking Image-Text Data at Scale

arXiv:2310.20550v3 Announce Type: replace-cross
Abstract: Large multimodal models demonstrate remarkable generalist ability to perform diverse multimodal tasks in a zero-shot manner. Large-scale web-based image-text pairs contribute fundamentally to this success, but suffer from excessive noise. Recent studies use alternative captions synthesized by captioning models and have achieved notable benchmark performance. However, our experiments reveal significant Scalability Deficiency and World Knowledge Loss issues in models trained with synthetic captions, which have been largely obscured by their initial benchmark success. Upon closer …

abstract arxiv benchmark captioning captions cs.ai cs.cl cs.cv cs.lg data diverse however image large multimodal models multimodal multimodal models noise performance scale studies success synthesized tasks text type web zero-shot