Consistency-guided Prompt Learning for Vision-Language Models

arXiv:2306.01195v2 Announce Type: replace
Abstract: We propose Consistency-guided Prompt learning (CoPrompt), a new fine-tuning method for vision-language models. Our approach improves the generalization of large foundation models when fine-tuned on downstream tasks in a few-shot setting. The basic idea of CoPrompt is to enforce a consistency constraint in the prediction of the trainable and pre-trained models to prevent overfitting on the downstream task. Additionally, we introduce the following two components into our consistency constraint to further boost the performance: enforcing …

abstract arxiv basic cs.cv few-shot fine-tuning foundation language language models prediction prompt prompt learning tasks type vision vision-language models