Differentially Private Knowledge Distillation via Synthetic Text Generation

arXiv:2403.00932v1 Announce Type: new
Abstract: Large Language models (LLMs) are achieving state-of-the-art performance in many different downstream tasks. However, the increasing urgency of data privacy requires LLMs to train with Differential Privacy (DP) on private data. Concurrently it is also necessary to compress LLMs for real-life deployments on resource-constrained devices or latency-sensitive applications. Differential privacy and model compression generally must trade off utility loss to achieve their objectives. Moreover, concurrently achieving both can result in even more utility loss. To …

abstract art arxiv cs.cl cs.cr cs.lg data data privacy deployments devices differential differential privacy distillation knowledge language language models large language large language models latency life llms performance privacy private data state synthetic tasks text text generation train type via