PromptSync: Bridging Domain Gaps in Vision-Language Models through Class-Aware Prototype Alignment and Discrimination

arXiv:2404.07520v1 Announce Type: new
Abstract: The potential for zero-shot generalization in vision-language (V-L) models such as CLIP has spurred their widespread adoption in addressing numerous downstream tasks. Previous methods have employed test-time prompt tuning to adapt the model to unseen domains, but they overlooked the issue of imbalanced class distributions. In this study, we explicitly address this problem by employing class-aware prototype alignment weighted by mean class probabilities obtained for the test sample and filtered augmented views. Additionally, we ensure …

abstract adapt adoption alignment arxiv class clip cs.cl cs.cv discrimination domain domains issue language language models prompt prompt tuning tasks test through type vision vision-language models zero-shot