Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning

arXiv:2404.00603v1 Announce Type: new
Abstract: We propose a generalized method for boosting the generalization ability of pre-trained vision-language models (VLMs) while fine-tuning on downstream few-shot tasks. The idea is realized by exploiting out-of-distribution (OOD) detection to predict whether a sample belongs to a base distribution or a novel distribution and then using the score generated by a dedicated competition based scoring function to fuse the zero-shot and few-shot classifier. The fused classifier is dynamic, which will bias towards the zero-shot …

abstract arxiv boosting cs.cv detection detectors distribution few-shot fine-tuning generalized language language models novel prompt prompt tuning sample tasks type vision vision-language models vlms