RegionGPT: Towards Region Understanding Vision Language Model

arXiv:2403.02330v1 Announce Type: new
Abstract: Vision language models (VLMs) have experienced rapid advancements through the integration of large language models (LLMs) with image-text pairs, yet they struggle with detailed regional visual understanding due to limited spatial awareness of the vision encoder, and the use of coarse-grained training data that lacks detailed, region-specific captions. To address this, we introduce RegionGPT (short as RGPT), a novel framework designed for complex region-level captioning and understanding. RGPT enhances the spatial awareness of regional representation …

abstract arxiv captions cs.cv data encoder image integration language language model language models large language large language models llms regional spatial struggle text through training training data type understanding vision vision language model visual vlms