RESTORE: Towards Feature Shift for Vision-Language Prompt Learning

arXiv:2403.06136v1 Announce Type: new
Abstract: Prompt learning is effective for fine-tuning foundation models to improve their generalization across a variety of downstream tasks. However, the prompts that are independently optimized along a single modality path, may sacrifice the vision-language alignment of pre-trained models in return for improved performance on specific tasks and classes, leading to poorer generalization. In this paper, we first demonstrate that prompt tuning along only one single branch of CLIP (e.g., language or vision) is the reason …

abstract alignment arxiv cs.cv feature fine-tuning foundation however language path performance pre-trained models prompt prompt learning prompts restore shift specific tasks tasks type vision