Generalizing CLIP to Unseen Domain via Text-Guided Diverse Novel Feature Synthesis

arXiv:2405.02586v1 Announce Type: new
Abstract: Vision-language foundation models like CLIP have shown impressive zero-shot generalization, but finetuning on downstream datasets can cause overfitting and loss of its generalization ability on unseen domains. Although collecting additional data from new domains of interest is possible, this method is often impractical due to the challenges in obtaining annotated data. To address this, we propose a plug-and-play feature augmentation method called LDFS (Language-Guided Diverse Feature Synthesis) to synthesize new domain features and improve existing …

abstract arxiv clip cs.cv data datasets diverse domain domains feature finetuning foundation language loss novel overfitting synthesis text type via vision vision-language zero-shot