Understanding the Multi-modal Prompts of the Pre-trained Vision-Language Model

arXiv:2312.11570v2 Announce Type: replace
Abstract: Prompt learning has emerged as an efficient alternative for fine-tuning foundational models, such as CLIP, for various downstream tasks. However, there is no work that provides a comprehensive explanation for the working mechanism of the multi-modal prompts. In this paper, we conduct a direct analysis of the multi-modal prompts by asking the following questions: $(i)$ How do the learned multi-modal prompts improve the recognition performance? $(ii)$ What do the multi-modal prompts learn? To answer these …

abstract analysis arxiv clip cs.cv fine-tuning foundational models however language language model modal multi-modal paper prompt prompt learning prompts tasks type understanding vision work