Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning

arXiv:2405.05615v1 Announce Type: cross
Abstract: Current solutions for efficiently constructing large vision-language (VL) models follow a two-step paradigm: projecting the output of pre-trained vision encoders to the input space of pre-trained language models as visual prompts; and then transferring the models to downstream VL tasks via end-to-end parameter-efficient fine-tuning (PEFT). However, this paradigm still exhibits inefficiency since it significantly increases the input length of the language models. In this paper, in contrast to integrating visual prompts into inputs, we regard …

arxiv cs.cl cs.cv cs.lg fine-tuning language memory prompting space type vision vision-language visual visual prompting