Understanding Optimal Feature Transfer via a Fine-Grained Bias-Variance Analysis

arXiv:2404.12481v1 Announce Type: cross
Abstract: In the transfer learning paradigm models learn useful representations (or features) during a data-rich pretraining stage, and then use the pretrained representation to improve model performance on data-scarce downstream tasks. In this work, we explore transfer learning with the goal of optimizing downstream performance. We introduce a simple linear model that takes as input an arbitrary pretrained feature transform. We derive exact asymptotics of the downstream risk and its fine-grained bias-variance decomposition. Our finding suggests …

abstract analysis arxiv bias bias-variance cs.lg data explore feature features fine-grained learn paradigm performance pretraining representation stage stat.ml tasks transfer transfer learning type understanding variance via work