Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

arXiv:2403.02969v1 Announce Type: new
Abstract: Multimodal Large Language Model (MLLMs) leverages Large Language Models as a cognitive framework for diverse visual-language tasks. Recent efforts have been made to equip MLLMs with visual perceiving and grounding capabilities. However, there still remains a gap in providing fine-grained pixel-level perceptions and extending interactions beyond text-specific inputs. In this work, we propose {\bf{AnyRef}}, a general MLLM model that can generate pixel-wise object perceptions and natural language descriptions from multi-modality references, such as texts, boxes, …

abstract arxiv beyond capabilities cognitive cs.cv diverse fine-grained framework gap interactions language language model language models large language large language model large language models llms mllms modal multi-modal multimodal multimodal large language model perception pixel tasks text type visual