On the Convergence of Differentially-Private Fine-tuning: To Linearly Probe or to Fully Fine-tune?

arXiv:2402.18905v1 Announce Type: new
Abstract: Differentially private (DP) machine learning pipelines typically involve a two-phase process: non-private pre-training on a public dataset, followed by fine-tuning on private data using DP optimization techniques. In the DP setting, it has been observed that full fine-tuning may not always yield the best test accuracy, even for in-distribution data. This paper (1) analyzes the training dynamics of DP linear probing (LP) and full fine-tuning (FT), and (2) explores the phenomenon of sequential fine-tuning, starting …

abstract arxiv convergence cs.ai cs.cr cs.lg data dataset fine-tuning machine machine learning math.oc optimization pipelines pre-training private data probe process public training type