VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning

arXiv:2406.14056v1 Announce Type: new
Abstract: Recent advances in Large Vision-Language Models (LVLMs) have significantly improve performance in image comprehension tasks, such as formatted charts and rich-content images. Yet, Graphical User Interface (GUI) pose a greater challenge due to their structured format and detailed textual information. Existing LVLMs often overly depend on internal knowledge and neglect image content, resulting in hallucinations and incorrect responses in GUI comprehension.To address these issues, we introduce VGA, a fine-tuned model designed for comprehensive GUI understanding. …

abstract advances arxiv assistant challenge charts cs.cv fine-tuning format gui hallucinations image images information language language models performance tasks textual through tuning type vision vision-language vision-language models