Pre-training Differentially Private Models with Limited Public Data

arXiv:2402.18752v1 Announce Type: new
Abstract: The superior performance of large foundation models relies on the use of massive amounts of high-quality data, which often contain sensitive, private and copyrighted material that requires formal protection. While differential privacy (DP) is a prominent method to gauge the degree of security provided to the models, its application is commonly limited to the model fine-tuning stage, due to the performance degradation when applying DP during the pre-training stage. Consequently, DP is yet not capable …

abstract arxiv cs.cr cs.lg data differential differential privacy foundation massive material performance pre-training privacy protection public public data quality quality data security training type