Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion

arXiv:2402.12195v1 Announce Type: new
Abstract: With the bloom of Large Language Models (LLMs), Multimodal Large Language Models (MLLMs) that incorporate LLMs with pre-trained vision models have recently demonstrated impressive performance across diverse vision-language tasks. However, they fall short to comprehend context involving multiple images. A primary reason for this shortcoming is that the visual features for each images are encoded individually by frozen encoders before feeding into the LLM backbone, lacking awareness of other images and the multimodal instructions. We …

abstract arxiv bloom context cs.cl diverse fusion images language language models large language large language models llm llms mllms multimodal multimodal content multiple performance prior reason tasks type via vision vision models