u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model

arXiv:2311.05348v3 Announce Type: replace
Abstract: Recent advancements in multi-modal large language models (MLLMs) have led to substantial improvements in visual understanding, primarily driven by sophisticated modality alignment strategies. However, predominant approaches prioritize global or regional comprehension, with less focus on fine-grained, pixel-level tasks. To address this gap, we introduce u-LLaVA, an innovative unifying multi-task framework that integrates pixel, regional, and global features to refine the perceptual faculties of MLLMs. We commence by leveraging an efficient modality alignment approach, harnessing both …

arxiv cs.cv language language model large language large language model llava modal multi-modal tasks type via