AutoFT: Learning an Objective for Robust Fine-Tuning

arXiv:2401.10220v2 Announce Type: replace-cross
Abstract: Foundation models encode rich representations that can be adapted to downstream tasks by fine-tuning. However, fine-tuning a model on one data distribution often degrades performance under distribution shifts. Current approaches to robust fine-tuning use hand-crafted regularization techniques to constrain the fine-tuning process towards the pretrained model. Yet, it is hard to specify how to adapt relevant characteristics of the foundation model during fine-tuning, as this depends on how the pre-training, fine-tuning, and test data distributions …

abstract arxiv cs.cv cs.lg current data distribution encode fine-tuning foundation however performance process regularization robust tasks type