Omniview-Tuning: Boosting Viewpoint Invariance of Vision-Language Pre-training Models

arXiv:2404.12139v1 Announce Type: new
Abstract: Vision-Language Pre-training (VLP) models like CLIP have achieved remarkable success in computer vision and particularly demonstrated superior robustness to distribution shifts of 2D images. However, their robustness under 3D viewpoint variations is still limited, which can hinder the development for real-world applications. This paper successfully addresses this concern while keeping VLPs' original performance by breaking through two primary obstacles: 1) the scarcity of training data and 2) the suboptimal fine-tuning paradigms. To combat data scarcity, …

abstract applications arxiv boosting clip computer computer vision cs.cv development distribution hinder however images language paper pre-training robustness success training training models type vision vision-language world