Model Reprogramming Outperforms Fine-tuning on Out-of-distribution Data in Text-Image Encoders

arXiv:2403.10800v1 Announce Type: new
Abstract: When evaluating the performance of a pre-trained model transferred to a downstream task, it is imperative to assess not only the in-distribution (ID) accuracy of the downstream model but also its capacity to generalize and identify out-of-distribution (OOD) samples. In this paper, we unveil the hidden costs associated with intrusive fine-tuning techniques. Specifically, we demonstrate that commonly used fine-tuning methods not only distort the representations necessary for generalizing to covariate-shifted OOD samples (OOD generalization) but …

abstract accuracy arxiv capacity cs.lg data distribution fine-tuning identify image paper performance samples text text-image type