Exploring the Transferability of Visual Prompting for Multimodal Large Language Models

arXiv:2404.11207v1 Announce Type: new
Abstract: Although Multimodal Large Language Models (MLLMs) have demonstrated promising versatile capabilities, their performance is still inferior to specialized models on downstream tasks, which makes adaptation necessary to enhance their utility. However, fine-tuning methods require independent training for every model, leading to huge computation and memory overheads. In this paper, we propose a novel setting where we aim to improve the performance of diverse MLLMs with a group of shared parameters optimized for a downstream task. …

abstract arxiv capabilities computation cs.ai cs.cv cs.lg every fine-tuning however independent language language models large language large language models mllms multimodal performance prompting tasks training type utility visual visual prompting