FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback

arXiv:2404.05046v1 Announce Type: new
Abstract: Large Vision-Language Models (LVLMs) have demonstrated proficiency in tackling a variety of visual-language tasks. However, current LVLMs suffer from misalignment between text and image modalities which causes three kinds of hallucination problems, i.e., object existence, object attribute, and object relationship. To tackle this issue, existing methods mainly utilize Reinforcement Learning (RL) to align modalities in LVLMs. However, they still suffer from three main limitations: (1) General feedback can not indicate the hallucination type contained in …

abstract arxiv cs.cl cs.cv current feedback fine-grained hallucination however image issue language language models object relationship tasks text type vision vision-language models visual