Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Models

arXiv:2311.12327v2 Announce Type: replace
Abstract: Visual grounding (VG) occupies a pivotal position in multi-modality vision-language models. In this study, we propose ViLaM, a large multi-modality model, that supports multi-tasks of VG using the cycle training strategy, with abundant interaction instructions. The cycle training between referring expression generation (REG) and referring expression comprehension (REC) is introduced. It enhances the consistency between visual location and referring expressions, and addresses the need for high-quality, multi-tasks VG datasets. Moreover, multi-tasks of VG are promoted …

arxiv cs.cv language language models training type vision vision-language vision-language models visual