SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant

arXiv:2403.11299v1 Announce Type: cross
Abstract: Recent advancements in the vision-language model have shown notable generalization in vision-language tasks after visual instruction tuning. However, bridging the gap between the pre-trained vision encoder and the large language models becomes the whole network's bottleneck. To improve cross-modality alignment, existing works usually consider more visual instruction data covering a broader range of vision tasks to fine-tune the model for question-answering, which are costly to obtain. However, the image contains rich contextual information that has …

abstract alignment arxiv assistant cs.ai cs.cl cs.cv cs.lg data encoder gap however language language model language models large language large language models llava network tasks type vision visual