Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance

arXiv:2404.00860v1 Announce Type: new
Abstract: Large-scale contrastive vision-language pre-trained models provide the zero-shot model achieving competitive performance across a range of image classification tasks without requiring training on downstream data. Recent works have confirmed that while additional fine-tuning of the zero-shot model on the reference data results in enhanced downstream performance, it compromises the model's robustness against distribution shifts. Our investigation begins by examining the conditions required to achieve the goals of robust fine-tuning, employing descriptions based on feature distortion …

abstract arxiv classification cs.cv cs.lg data fine-tuning guidance image language performance pre-trained models random reference results robust scale tasks text training type vision zero-shot