Selective Pre-training for Private Fine-tuning

Suppose we want to train text prediction models in email clients or word processors. These models, which serve billions of predictions per hour, must preserve the privacy of user data and adhere to specific model size constraints to meet memory, inference time requirements, and to reduce inference cost. Building small, fast, and private domain-specific language models is a thriving area of research. In this work, we show that a careful pre-training on a {\em subset} of the public dataset that …

building constraints cost cs.cr cs.lg data domain email fine-tuning hour inference language memory per prediction prediction models predictions pre-training privacy processors reduce requirements serve small text train training user data word