Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation

arXiv:2403.06946v1 Announce Type: new
Abstract: Large vision-language models (VLMs) like CLIP have demonstrated good zero-shot learning performance in the unsupervised domain adaptation task. Yet, most transfer approaches for VLMs focus on either the language or visual branches, overlooking the nuanced interplay between both modalities. In this work, we introduce a Unified Modality Separation (UniMoS) framework for unsupervised domain adaptation. Leveraging insights from modality gap studies, we craft a nimble modality separation network that distinctly disentangles CLIP's features into language-associated and …

arxiv cs.cv domain domain adaptation merge type unsupervised