Anchor-based Robust Finetuning of Vision-Language Models

arXiv:2404.06244v1 Announce Type: new
Abstract: We aim at finetuning a vision-language model without hurting its out-of-distribution (OOD) generalization. We address two types of OOD generalization, i.e., i) domain shift such as natural to sketch images, and ii) zero-shot capability to recognize the category that was not contained in the finetune data. Arguably, the diminished OOD generalization after finetuning stems from the excessively simplified finetuning target, which only provides the class information, such as ``a photo of a [CLASS]''. This is …

abstract aim anchor arxiv capability cs.cv data distribution domain finetuning images language language model language models natural robust shift type types vision vision-language models zero-shot